Credibility: Robust Evaluation Models

Question 1

Credibility

Answer 1

Metrics Performance Evaluation: How evaluate?
Methods for Performance: How to obtain reliable?
Methods for Comparison: How to compare?
Model Selection: Which model?

Question 2

Metrics Performance Evaluation: How evaluate?

Answer 2

Residual Sum of Squares (RSS): sum(yi - sum(wjhj(xi))^(2))
R2: TSS = sum(yi - y)^2 R2 = 1 - RSS/TSS
MSE = (1/N)sum(yi*-yi)^2
RMSE = sqrt(MSE)

Question 3

Metrics Performance Evaluation: Classification

Answer 3

TP - FN
FP - TN

Accuracy = TP + TN / (ALL)
Cost(X): evaluate, or guide the search or to build model (decision trees)

Precision: positive that are actually positive. TP/(TP+FP)
Recall: positive classified w.r.t existing good doc. TP/(TP+FN)

Question 4

Methods for Performance: How to obtain reliable?

Answer 4

Training, Validation and Testing

Holdout: 2/3 - 1/3
Random Subsampling: repeated holdout.
Cross Validation: k disjoints subsets. k-fold. Leave one out.
Stratified sampling: each class represented with approx equal proportions.

Bootstrap: sampling with replacement.
0.632 bootstrap: probability of ending up in test data is: 0.368
error: 0.632etest + 0.368etrain
repeat several times.

Question 5

Methods for Comparison: How to compare?

Answer 5

Paired t-Test using k-fold Crossvalidation
Generate k folds and for each compute performance of model A and B
di = perf ai - perf bi
mean = (1/k)sum(di)
std=sqrt((1/k)sum(di-mean)^2)
Two hypothesis: H0: mean = 0, Ha: mean != 0

Question 6

Methods for Comparison: Multiple Testing

Answer 6

0.05 threshold in 20 different observations.
0.95^20 = 0.358
at least one mistake = 1 - 0.358 = 0.642

Bonferroni Correction
	Tests are independent
	Divide p-value by number of tests
	0.05/20 = 0.0025
	at least one mistake = 1 - 0.9512 = 0.0488

Question 7

Methods for Comparison: Probabilistic Classifiers

Answer 7

Logistic Regression returns a probability
P(yi|xi) threshold of 0.5
Threshold of 0.75 for one label.
Higher Threshold, more precision, low recall
Lower Threshold, less precision, more recall

Question 8

Methods for Comparison: Precision-Recall Curves

Answer 8

Precision as function of recall varying threshold values.

Best classifier would be the one with precision always equal to one

Question 9

Methods for Comparison: Receiver Operating Characteristicc ROC

Answer 9

TPR = TP/(TP+FN)
FPR = FP/(FP+TN)

(0,0) everything negative
(1,1) everything positive
(0,1) ideal

No model consistently outperform other.

Question 10

Methods for Comparison: Lift Charts

Answer 10

Measure of effectiveness of predictive model
Ratio between with and without predictive model
Cumulative Gains and Lift Charts are visual aids
Greater area, better the model.
100.000 response rate is 0.4% that is 400 response.

Question 11

Model Selection: Which model?

Answer 11

Occam’s Razor: best theory is the smallest one that describes all the facts.
No free lunch Theorem: no favor one.