4 Backpropagation

Question 1

What is SGD?

Answer 1

Stochastic gradient descent

Question 2

What is backpropagation?

Answer 2

A method for computing gradients in neural nets

Question 3

What is (mini) batch gradient descent?

Answer 3

A simple strategy for optimization

Question 4

What is the expected loss

Answer 4

E_x,y~phat_data L(yhat,y)

Expectation meansaverage over training examples

Question 5

How do we use ML to optimize for w

Answer 5

If we know the distribution of our model:
Log likelihood

W_ML = argmax FnzyL(w) = argmax sum(m,i=1) log p(yi|xi)

Question 6

What is p(x|y) in ML

Answer 6

Least squares it is Gaussian
For classification is is multinomial

Question 7

What is cross-entropy

Answer 7

Statistical divergence between the outputs of the model and the examples in the training set.

Maximizing LL is equilivent to minimizing the CE

• sum(m,i=1) log p(yi|xi)

Question 8

What is cross-entropy also called?

Answer 8

Loss of the cost function

Question 9

How do we calculate loss in prqactice?

Answer 9

Take average (1/m) over the presented examples

We specify f that tries to predict p(y|x) output yhat = f(x)
MSE loss: L(yhat,y) = sum(m,i=1) ||yi -yihat||^2
Log loss: L(y,yhat) = sum(m,i=1) yi log yihat

Question 10

How do we do backpropagation?

Answer 10

Backpropergate the gradiants starting fromthe loss.

Question 11

Draw a computational 2 layer grapgh for a NN

Answer 11

h = f1(a1) = f1(w1 x+ b1)
yhat = f2(a2) =f2(w2h + b2)

draw your grapgh on paper

Question 12

Gradiant descent

Question 13

Batch gradiant descent