W07 Validation and Evaluation

Question 1

Forecast Evaluation

2 possibilities

Answer 1

• compare to actual observations (beware of self fulfilling prophecies)
• compare to naive

Question 2

Error Measures

Answer 2

Absolute
Percentage
Scaled

Question 3

Self-fulfilling forecast - negative consequence

Answer 3

buy-down

Question 4

Classification Performance

Answer 4

Recall (Sensitivity)
correctly assigned / actually in class

Precision
actually in class and assigned / assigned to class

Specificity
how many (share of) not selected are actually not true?

Question 5

Error rate across categories

Answer 5

average or weighted average or importance-weighted

Question 6

Comparing error rates

Answer 6

training vs validation vs test set

expected error vs observed error vs benchmark approach

Question 7

Possible Benchmarks

Answer 7

statistically expected error rate
naive rules
expert assignment

Question 8

Benchmark Factors beyond accuracy

Answer 8

effort
reliability
acceptance

Question 9

Data Set split

Answer 9

Training Set: build tree
Validation Set: prune tree
Test Set: evaluate tree’s predictions

Question 10

Testing: Hold out 1

Answer 10

k-fold cross validation

1 split data into k partitions of equal size

2 use k-1 for training
3 use k for evaluation

4 repeat k times
5 average the results

Question 11

Testing: Hold out 2

Answer 11

Bootstrap
alternative to cross validation, for small data sets

n is original data set size

draw n instances with replacement (same can be drawn multiple times)

This is the training set.

Never drawn instances are test set.

Question 12

Lift Factor

Answer 12

What increase in accuracy does my prediction promise?

Gives ratio, not absolutes. Helpful for cost-benefit analysis.

Question 13

Lift Chart

Answer 13

when classification is probabilistic

compute lift factor when increasing sample size, possibly comparing to increase in cost caused by increasing the sample size