lecture 8: optimisation and gradient descent

Question 1

in general, an objective function can be described by

Answer 1

cost function = loss function(learning model) + regularisation
this as a whole, plus the optimisation routine used to solve, makes up the building blocks of ML algorithms

Question 2

what is gradient descent

Answer 2

moving along the model in the direction where gradient is decreasing according to some learning rate eta(𝜂) until some convergence criteria is reached, to find an estimate of the local minima

Question 3

what are some possible convergence criteria

Answer 3

1. set maximum iteration k
2. check percentage/absolute change in C below a threshold
3. check percentage/absolute change in w below a threshold

Question 4

why can gradient descent only find the local minima?

Answer 4

because gradient = 0 at local minima, so w cannot change after that

Question 5

what does increasing the learning rate do

Answer 5

it increases the rate at which w converges, too high of an eta can cause the function to repeatedly overshoot the local minima

Question 6

what is the purpose of loss functions

Answer 6

different loss functions encode the penalty for prediction when the true value is yᵢ

Question 7

what is binary loss function

Answer 7

the product of f(x,w) and yᵢ
if the sign is positive, classification is correct and and loss is 0, if sign is negative then classification is wrong and loss is 1

Question 8

binary loss is not differentiable, what are 2 other possible loss functions to use?

Answer 8

hinge loss and exponential loss
hinge loss = max(0, 1 - prediction)
exponential loss = exp(-prediction)

Question 9

what does the sigmoid function do

Answer 9

used for classification, maps pᵢᵀw(which is between ∞ and -∞) to between 1 and -1
f(x,w) = σ(pᵢᵀw)
σ(a) = 1/(1 + e⁻ª)