Model Evaluation

Question 1

Formula for Accuracy

Answer 1

True Positive + True Negative / Number of Items

Question 2

Formula for Precision (p)

Answer 2

p = TP / TP + FP

Question 3

Formula for Recall (r)

Answer 3

r = TP / TP + FN

Question 4

Formula for F-Measure

Answer 4

Precision = p
Recall = r

fm = 2rp / r + p

Question 5

compare the accuracy of the classifier with a random classifier.

Answer 5

Kappa Statistics

Question 6

was developed in 1950 for signal detection theory.’

Works only for binary classification.

Answer 6

Receiver Operating Characteristic (ROC)

Question 7

to estimate performance of classifier on previously unseen data.

Answer 7

Purpose of Model Evaluation

Question 8

reserve k% for training and (100 - k) % for testing.

Answer 8

Holdout

Question 9

partition data into k disjoint subset.

Answer 9

Cross Validation

Question 10

train on k-1 partition, test on the remaining one.

Answer 10

K- Fold

Question 11

shows how accuracy on unseen examples changes with varying training sample size.

Answer 11

Learning Curve

Question 12

many algorithms allow choices for learning.

Answer 12

Hyperparameters

Question 13

3 STEPS IN TRAINING THE MODEL

Answer 13

1. Train
2. Model Selection
3. Test

Question 14

learn models on the training data using different hyperparameters.

Answer 14

Train

Question 15

evaluate the models using the validation data and choose the hyperparameters with the best accuracy.

Answer 15

Model Selection

Question 16

test the final model using the test data.

Answer 16

Test

Question 17

3 TYPES OF CLASSIFICATIONS ERRORS

Answer 17

• Training Errors
• Test Errors
• Generalization Errors

Question 18

errors committed on the training set.

Answer 18

Training Errors

Question 19

errors committed on the test set.

Answer 19

Test Errors

Question 20

expected errors of a model over random selection of records from same distribution.

Answer 20

Generalization Errors

Question 21

is when a model is too simple, both training and test errors are large.

Answer 21

Underfitting

Question 22

when model is too complex, training error is small, but test error is large.

Answer 22

Overfitting

Question 23

2 REASONS FOR OVERFITTING

Answer 23

1. Not enough training data
2. High model complexity

Question 24

2 MODEL SELECTION FOR DECISION TREE

Answer 24

1. Pre-Pruning (Early Stopping Rule)
2. Post-Pruning

Question 25

stops the algorithm before it becomes a fully grown tree.

Answer 25

Pre-Pruning (Early Stopping Rule)

Question 26

grow decision tree to its entirety.

Answer 26

Post-Pruning