Testing and validating

Question 1

what is the generalisation error?

Answer 1

the error rate on new cases, it tells you how well your model will perform on instances it has never seen before

Question 2

what does a low training error + high generalisation error mean?

Answer 2

you model is overfitting the data

Question 3

what is meant by testing and validating?

Answer 3

trying out a model on new cases to see how well the model generalizes

Question 4

what is a validating set?

Answer 4

a second hold out set used when you adapted a model to suit a test set, and it doesn’t work as well in production.

Question 5

what do you do with the validation set?

Answer 5

you train various hyperparameters that work best on the training set and then you select the model and hyperparameters that performs best on the validation set.

Question 6

what is cross validation?

Answer 6

the training set is split into k folds and each model is trained against a different combination of these subsets and validated against the remaining parts