Practical Issues of ML

Question 1

How would you determine the size of a validation set?

Answer 1

The validation set needs to be large enough to detect the performance difference between two or more models, but not necessarily much larger.

Question 2

What are some ways to improve the Bias Error?

Answer 2

Improve feature engineering e.g. outlier removal

Improve model architecture or try another method

Reduce regularisation

Increase the model size

Question 3

What are some ways to improve the Variance Error?

Answer 3

Add Regularisation or decrease the model size

Improve feature selection e.g. reducing dimensions, picking subsets, etc…

Add more training data

Question 4

What are some ways to improve the Mismatch Error?

Answer 4

Understand the difference between training and testing sets

Add more training data that is similar to the test cases

Question 5

What is the best way to evaluate a model?

Answer 5

Understand the key aim of the task, and choose the most appropriate single measure for the given task.

If multiple metrics are needed, order their priority