evaluating models Flashcards
How do we rely on models?
we need to know how good they will be at their job before relying on them
Why one must evaluate classifier performance?
-multiple methods are available to classify or predict
-for each method, multiple options are available
-to choose the best model, one needs to assess each model’s performance, relative to others
What is the misclassifcation effors?
-error
-error rate
-accuracy
what is error under misclassifcation error?
classifying a record as belonging to one class when it belongs to another class
What is error rate under misclassification error?
percent of misclassifed records ou of the total records in the valdidation data
what is accuracy under misclassification error
accuracy = 1 - error rate
What is confusion matrix
it summarizes binary classification error
what is accuracy
the overall probability the model will predict actual outcomes correctly TP+ T N / (P+N) slide 15 of eval. models
what is true positive rate
its the sensitivity
- the proporition of events correctly predicted
- of all the times the event occurred, what is the proporitoin of times it was precicted to occur, the model’s hit rate?
What is the formula of True Positive Rate?
TP/(TP+FN)
WHAT IS THE INVERSE OF SENSITIVITY ?
miss rate
What is a true negative rate?
- the proportion of non-events correctly predicted
- of all the times ethe event did not occur, what was the proportion of times it was predicted to not occur?
What is positive predicitive value
the proption of postive event predictions that were correct
- more false positives erode precision, many false claims the event will occur means your model is less discerning when predicting the event
what is the relationship between sensitivity and precision?
if an event occurs 10/1000 cases then its rare
- if the model predicts 9 of the 10 , it is highly sensitive to the phenomen
-yet if the model also predicts the event 900 more times then the model is not precise
not all errors are equal
doctor example, false exam of meningitis or dying at tnight because doctor mistreated u
