Networks

Question 1

What are two alternatives to the Relu function?

Answer 1

ELU(Exponential linear units) and leaky Relu.

Question 2

what is the difference between SGD with momentum and SGD with Nestrov momentum?

Answer 2

Nestrov momentum adds the velocity to parameter before computing gradients

Question 3

What is the idea behind Adagrad?

Answer 3

Adagrad reduces the learning rate of gradient dimensions with large square value.

Question 4

What is the idea behind RMS prop?

Answer 4

Adagrad reduces the learning rate of gradient dimensions with large running mean square value.

Question 5

What is the idea behind Adam optimizer?

Answer 5

Combines SGD with momentum and RMS prop.

Question 6

Name some regularizers

Answer 6

L1, L2, Early stopping, Dropout, constrain max norm, data augmentation

Question 7

Name two commonly used initialization methods for weights in a Neural network

Answer 7

w = N(0, 1/sqrt(N)) or Xavier: U(-1/sqrt(N), 1/sqrt(N))

Question 8

What is the formula for batch normalization?

Answer 8

z_hat = (z - mu)/sigma
z_new = y*z_hat + b, where y and b are learnt parameters.

Question 9

What is the ouput size after a convolutional layer

Answer 9

out = (in + 2*pad - filter_size)/stride + 1

Question 10

Why are deep networks harder to train and how can we solve this?

Answer 10

1. Vanishing gradient:
   - Relu
   - Good initializaiton
   - Auxiliary classifiers
2. Covariate shift:
   - Batch normalization

Question 11

What is the idea behind the inception net?

Answer 11

Each layer tries several filter sizes and the networks “learns” the best filter size