Capacity, Overfitting And Underfitting

Question 1

The ability to perform well on previously unobserved inputs is called

Answer 1

Generalisation

Question 2

Generalisation error

Answer 2

Aka test error

For linear regression (image)

Question 3

Training error (linear regression)

Answer 3

Question 4

Data generating process

Answer 4

Making IID assumptions of training and test data collectively

We assume we are drawing test and train data from sample distribution (data generating distribution demoted p_data )

Question 5

caveat to data generating process in ML

Answer 5

We do not fix parameters ahead of time

We sample training set then use it to choose parameters to reduce training set error THEN sample test set

Therefore, E_{test error} >= E_{training error}

(Where without this process they would be equal as they’re from same dist)

Question 6

Factors determining how well an ML algorithm performs

Answer 6

1) make training error small (large error is underfitting)

2) make gap between training error and test error small (large gap is overfitting)

Question 7

How do we control overfitting/underfitting

Answer 7

Altering capacity

Question 8

Capacity def

Answer 8

(Informally) a model’s capacity is its ability to fit a wide variety of functions

Low capacity models may struggle to fit training set

High capacity may over fit by ‘memorising’ training set

Question 9

Hypothesis space

Answer 9

The set of functions that the learning algorithm is allowed to select as a solution

Altering this is one way of altering capacity

Question 10

Increasing capacity of linear regression example

Answer 10

Adding polynomial terms of the inputs does not require polynomial terms in the output

Question 11

Draw and explain the underfitting over fitting diagram

Answer 11

Self explanatory diagram

Question 12

Def regularization

Answer 12

Any modification that we make to a learning algorithm to intent reduce its generalisation error but not its training error

Question 13

K fold cross validation algorithm

Answer 13

Question 14

Point estimator

Answer 14

Any function of the parameters

Question 15

Frequentlist perspective on statistics

Answer 15

Question 16

Weight decay

Answer 16

Form of regularisation for linear regression

(Image is quantity to minimise)

Question 17

Asymptotically unbiased

Answer 17

Question 18

Variance of estimator and standard error

Answer 18

Question 19

Navigating bias variance trade off

Answer 19

(Can use cross validation)

Or

Question 20

Show how bias and variance relate to over and underfitting in a diagram

Answer 20