Bias Variance Trade off and Inductive Bias

Question 1

Reducing bias when having a large variance:

Answer 1

When you have a large complex model and training data S is small, small changes to S lead to large changes in the model. This is a risk of overfitting.

Question 2

Reduce variance having a large bias:

Answer 2

When you have a restricted, simple model, and the training data is small, small changes to it don’t lead to large changes in the model. This leads to a risk of underfitting (strong generalisation without sufficiently fitting the model)

Question 3

What do machine learning algorithms need to be able to generalise to new data?

Answer 3

It needs bias (inductive bias)

Question 4

What are the two sources of bias?

Answer 4

Representation bias

Search bias

Question 5

What is inductive bias?

Answer 5

It is a criterion (preference or assumption) different from consistency with the data, used to favour a hypothesis over another.

Question 6

What must inductive algorithms have?

Answer 6

Inductive bias

Question 7

What is the approximation error?

Answer 7

How much error we have because we restrict ourselves to a specific class of models (i.e., how much inductive bias we have)

Question 8

What is estimation error?

Answer 8

The training error is only an estimation of the true error.

Question 9

What does every classification algorithm have?

Answer 9

Inductive bias

Question 10

Is bias suitable for all data?

Answer 10

Yes and no. Every bias is suitable for some data sets and unsuitable for others.

Question 11

What is the best way to find the best result?

Answer 11

Trying several different classification algorithms and using the best result (called the Toolbox Approach)