Chapter 5

Question 1

Prediction

Answer 1

Average Error, MAPE (
Mean Absolute Percentage Error
), RMSE 
(Root-Mean-Square-Error)
, 
Validation Data

Question 2

Classification

Answer 2

Classification matrix, specificity, sensitivity

Question 3

ROC (Receiver Operating Characteristic)

Answer 3

to assess performance at different cutoff values

Question 4

Detect overfitting

Answer 4

compare validation to training data:

some differences expected, extreme differences may indicate overfitting

Question 5

Naïve rule

Answer 5

classify all records as belonging to the most prevalent class
benchmark:  we hope to do better than that
Using external predictor info should outperform naïve rule

Question 6

Exception to Naïve rule

Answer 6

when goal is to identify high-value but rare outcomes, we may do well by doing worse than the naïve rule (see “lift” – later)

Question 7

There are various performance measures comparing to the naïve rule

Answer 7

For example: multiple R squared, measures classifier fit to naïve rule.

The equivalent to using the naïve rule for classification is y^ (the sample mean)

Question 8

Lift Chart for Predictive Error

Answer 8

Y axis is cumulative value of numeric target variable (e.g., revenue), instead of cumulative count of “responses”

X axis is cumulative number of cases, sorted left to right in order of predicted value

Benchmark is average numeric value per record, i.e. not using model (aka The Naïve Rule)

Question 9

Misclassification error

Answer 9

Error = classifying a record as belonging to one class when it belongs to another class.

Error rate = percent of misclassified records out of the total records in the validation data

Question 10

“High separation of records”

Answer 10

means that using predictor variables attains low error

Question 11

“Low separation of records”

Answer 11

means that using predictor variables does not improve much on naïve rule

Question 12

Confusion Matrix

Answer 12

actual class
predicted

Question 13

Accuracy

Answer 13

1 – err

Question 14

Cutoff Table

Answer 14

cut off is .50 , so everything above should be 1 and everything below should be 0. any records that are otherwise, it counted as misclassification

Question 15

When One Class is More Important

Answer 15

we are willing to tolerate greater overall error, in return for better identifying the important class for further attention

Question 16

Sensitivity

Answer 16

The ability to detect important class members correctly

Question 17

Specificity

Answer 17

ability to rule out C0 members classified correctly

Question 18

False positive

Answer 18

% of predicted “C1’s” that were not “C1’s”
false alarm,
indicates a given condition exists, when it does not.

Question 19

False negative

Answer 19

% of predicted “C0’s” that were not “C0’s”

indicates a given condition not exists, but it really does

Question 20

Lift and Decile Charts: Goal

Answer 20

Useful for assessing performance in terms of identifying the most important class
The goal is to obtain a rank ordering among the records according to their estimated
probabilities of class membership
Compare performance of DM model to “no model, pick randomly”

Question 21

Decile Chart

Answer 21

In “most probable” (top) decile, model is twice as likely to identify the important class compared to avg. prevalence.

Question 22

Lift vs. Decile Charts

Answer 22

Decile chart does this in decile chunks of data
Y axis shows ratio of decile mean to overall mean

Lift chart shows continuous cumulative results
Y axis shows number of important class records identified