Lecture 4 - Training CNNS

Question 1

Regression

Answer 1

A set of processes for modelling the relationship between an output and input features

Question 2

Classification

Answer 2

Problem of classifying inputs

Question 3

How to measure regression performance

Answer 3

Loss Function like Mean Absolute Error or Mean Squared Error,…

Question 4

Loss Function: Value size when it is well classified vs not

Answer 4

well classified = small value
else big value

Question 5

Negative Log Likelihood Loss

Answer 5

Lp = -Log(p)

Question 6

Softmax

Answer 6

Normalise network output tio a probability distribution over predicted output classes

exp(xi) divided by the sum of all j where exp(xj)

Question 7

Cross entropy loss

Answer 7

-Log(softmax)

Question 8

Optimisation

Answer 8

Process to find best weight to minimise loss function

Question 9

Gradient Descent

Answer 9

Gradient direciton and step size. “Walk” towards minima.

Question 10

Learning Rate

Answer 10

Step size.

Too low = no progress
too high = instability and never converges

Question 11

Stochastic Gradient Descent

Answer 11

each w = w - a * gradient value

a here is alpha (learning rate)

see slides for full equation

Question 12

SGD Weight Decay

Answer 12

Used to prevent the weight being too big

w <- w - a(deriv(w)L(x,w)-yw) + p[last update]
a is alpha (learning rate) and y is gamma. yw is weight decay regularisation

Question 13

SGD Momentum

Answer 13

p[last update]

where p is a number less than 1 (meaning p * update i think?)

Question 14

“Gradient Value” equation. Using weights, inputs and derivative of..?

Answer 14

sum(i)(deriv(w)*L(xi,w))