SGD

Question 1

What optimizer for non linear function of θ?

Answer 1

Gradient based optimizer

Question 2

How to compute the gradient of a NN?

Answer 2

compute the partial derivative of the loss with respect to all parameters θk (i.e., the weights and the biases of all layers):

Question 3

What does Splitting the training set into B minibatches do?

Answer 3

• reduces the computation cost of one gradient by a factor of B
• increases the standard deviation on the gradient estimate by a factor of √B only.

More iterations, but fewer epochs (hence smaller total
computation cost).