Regularisation

Question 1

Goal of regularisation in NNs

Answer 1

Take a model that includes the generating process but also many other possible generating processes

And

Help it match the true data generating process

Question 2

Denote regularized objective function

Answer 2

Where:
α € [0, inf)
Ω(θ) is norm penalty

Question 3

Normal practise where selecting parameter norm penalty for NNs

Answer 3

Choose Ω to penalise only weights of affine transformations at each layer, leave biases as they typically require less data than weights to fit accurately

Largely because weights concern interactions of variables, biases concern single variables

Also, including biases can introduce a significant amount of underfitting

Question 4

w

Answer 4

Refers to all weights that should be affected by norm penalty while θ denotes all parameters including both w and unregularized parameters

Question 5

L² called?

Answer 5

Partner norm penalty, AKA weight decay, RIDGE regression or Tikhonov regression

Question 6

Show how L² works on gradient of objective function (let θ = w for simplicity)

Answer 6

Question 7

Further simplify by making quadratic appromation of objective function in neighbourhood of value of weights that obtains minimal inregularized training costs

Answer 7

If objective function is truly quadratic then approximation is prefect and given by

Question 8

L¹ regularisation term

Answer 8

Question 9

Give regularized objective function and corresponding gradient for L¹ (for SLRM)

Answer 9

Question 10

Describe difference in grad of objective function from L² to L¹

Answer 10

Regularization contribution of gradient no longer scales linearly with each wi

Instead it is a constant factor * sign(wi)

Therefore we won’t necessarily see clean algebraic solutions to quadratic approximations of (unregularized) objective function

Question 11

Taylor series expansion of grad of cost function for SLRM L¹

Answer 11

We are assuming the regularised cost function is a truncated Taylor series expansion of a more complicated model

Question 12

Decompose quadratic approximation of L¹ regularised objective function for SLRM

Answer 12

(We assume hessian is diagonal and each value on trace is pos for simplicity)

Question 13

Solution to

Answer 13

Question 14

Sparsity in context of regularization

Answer 14

Having parameters with optimal value of 0

Question 15

Solution to L² regularisation for SLRM (with θ = w)

Answer 15

7.13

Question 16

Motivate feature selection using L¹

Answer 16

(Image) therefore sparsity induced by L¹ can be used as a feature selection mechanism

Question 17

Log laplace

Answer 17