C2W3 Hyperparameters Tuning

Question 1

Tuning order

Answer 1

1. Learning rate
2. Beta(momentum term), mini batch size, hidden units
3. Number of layers, learning rate decay
4. ADAM parameters (beta1 0.9, beta2 0.999, epsilon 10-8)

Question 2

How to tune learning rate

Answer 2

Use logarithmic scaling (10^-4, … 10^0)

Question 3

Pandas vs caviar approach to training model

Answer 3

Pandas: 1 model at the time, caviar: multiple models at the time

Question 4

Batch normalisation

Answer 4

Normalising values deep in hidden layers, normalise mean and variance of inner Z.
This eliminates usage of B because of mean computing, instead Beta is used

Question 5

Batch norm at test time

Answer 5

Save M and Alpha for each mini batch (when train).
Compute exponential weighted average of them. (Estimated)

At test use it for scaling hidden units values when evaluating test example

Question 6

Soft Max regression

Answer 6

Soft Max layer is the latest layer in NN, with number of units equal to number of classes.
You apply softmax activation function to Zs of this layer

Question 7

Deep learning frameworks

Answer 7

Caffe/Caffe2
CNTK
DL4J
Keras
Lasagne
mxnet
PaddlePaddle
TesorFlow
Theano
Torch