Evaluate Classifiers Flashcards
Why would you need evaluate classifiers?
To help choose optimal method and parameters
What are 3 causes to overfitting?
- Too many variables
- Excessive model complexity
- Data leakage
What is the consequence of launching an overfit model?
Deployed model will not generalized
What is the formula for accuracy rate?
Accuracy rate = (# of correct classification) / (# of records in datatset)
when is accuracy alone is not a sufficient metric?
For imbalance classification problems
Along the lines of confusion matrix, what is the formula for accuracy?
Accuracy = (TP+TN) / (P + N)
What is the 2 step process for cutoff value for classification?
- Compute probability of belonging to positive class
- Compare cutoff value and classify
Along the lines of confusion matrix, what is the formula for precision?
Precision =
(TP) / (TP + FP)
Along the lines of confusion matrix, what is the formula for False Discovery?
FDR = 1 - Precision
Along the lines of confusion matrix, what is the formula for False Omission Rate?
FOR
= (FN) / (TN + FN)
Along the lines of confusion matrix, what is the formula for Recall?
Recall = (TP) / (TP + FN)
Along the lines of confusion matrix, what is the formula for False Negative Rate?
False Negative Rate = 1 - Recall
Along the lines of confusion matrix, what is the formula for false positive rate?
FPR = (FP) / (FP +TN)
What does accuracy measure overall?
Correctness
What type of data is accuracy good to use with?
Balanced datasets
When is accuracy misleading?
When working with imbalance classes
What does precision focus on?
Positive prediction
When false positives are costly, what is a good metric to use?
Precision
What does the ROC Curve & AUC evaluate?
Model performance at different thresholds
What does a higher AUC mean?
Better discrimination between classes
What does the ROC curve and AUC help find?
Optimal balance between true positive and false positive
What does the lift & Gain chart help asses?
How well the model ranks and prioritizes high value cases
What does the lift show?
Improvement over random selection
What does the gains help visualize?
How well the model capturees true positives early
Which evaluation metric is good for fraud detection, spam filter?
Precision
Which evaluation is good for medical diagnosis and security alerts?
Recall
