Supervised learning and Convolutional Networks

Question 1

What does SGD stand for? Why do we use this approach and not a full approach? Mention at least 3 variants of SGD

Answer 1

SGD = Stochastic gradient descent
Calculating gradients in every direction is too computationally expensive / not necessary (exploration).
1. SGD with momentum
2. RMSprop
3. Adam

Question 2

Mention three strategies for architecture search

Answer 2

1. Random search
2. Genetic algorithms
3. Reinforcement learning

Question 3

Mention two learned neural architecture search networks and explain their difference.

Answer 3

NAS1 & NAS2.
NAS1 has a fixed layering form (conv2d, batchnorm, relu) with parameters: conv2d params, inputs to layer (skip connections).
NAS2 has a fixed overall architecture (CIFAR10/ImageNet), but wants to learn the cell structure of a “normal” and “reduction” cell.
The two basically (ish) handles and assumes the inverse of one another.

Question 4

What is dilation rate?

Answer 4

How large the gaps are between elements in a feature map on which we apply a convolution filter.

Question 5

What is another word for L2 regularization and why do we use it?

Answer 5

Least squared. We use it to minimize the complexity of the parameters in the model.

Question 6

Mention ways to augment text and image data

Answer 6

Image: Mirroring, random crop, scale, aspect ratio, lightning
Text: Synonym insert, back-translation (google translate)