Optimization

Question 1

What are the hyperparameters in a gradient descent?

Answer 1

Weight initialization method
Number of steps before the algo stops
Learning rate per iterations

Question 2

What is batch gradient descent?

Answer 2

Batch gradient descent updates the model after the loss for each example in the training set is computed

Question 3

What is stochastic gradient descent?

Answer 3

Approximate sum of loss function using a minibatch of examples 32/64/128 common

Question 4

What new hypermeters arising from SGD?

Answer 4

Batch size and data sampling. Batch size does not matter too much. Make it as big as it can fit your hardware.

Data sampling : Draws from random. Does not matter too much as well esp for computer vision.

Question 5

What is high condition number?

Answer 5

High condition number usually means that the matrix is almost non-invertible. For SGD, it may lead to jitter and slow progress.

Question 6

What is problem with SGD?

Answer 6

It is stuck at local minimum or saddle point in which gradient is zero. Saddle point in which in some direction it is increasing and another direction it is decreasing.

Question 7

What other problem with SGD?

Answer 7

Gradient comes from minibatches so they can be noisy