College 5

Question 1

What is the objective of gradient descent?

Answer 1

Move 𝚹 (parameters) in the direction that minimizes the loss function J(𝚹)

Question 2

How does batch gradient descent work?

Answer 2

Updates the parameters 𝚹 by

calculating the gradients using the whole dataset.

Question 3

How does stochastic gradients descent work?

Answer 3

Updates the parameters 𝚹 by calculating the gradients using randomly selected samples (one at a time).

Question 4

What are the advantages of batch gradient descent?

Answer 4

less noisy, more precise, smoother path

Question 5

What are the advantages of stochastic gradient descent?

Answer 5

computationally fast, easier to fit in memory

Question 6

How does mini-batch gradient descent work?

Answer 6

Updates the parameters 𝚹 by calculating the gradients using “small”
batches of samples (somewhere between Stochastic and Batch Gradient Descent).

Question 7

Name 6 optimisers

Answer 7

• SGD
• momentum
• NAG
• Adagrad
• Adadelta
• RMSprop

Question 8

What are the limitations of (mini-batch stochastic gradient descent?

Answer 8

● It might get stuck in local minima.
● Choosing a proper learning rate can be difficult.
● The same learning rate applies to all parameter updates.

Question 9

What is the purpose of exponentially weighted averages?

Answer 9

To filter/smooth noisy (temporal) data to reduce the chance of getting stuck in a local minima

Question 10

Why is the filtered version of a noisy signal shifted to the right?

Answer 10

You take into account the previous timestep of your average and add a bit of the function you want to filter and it take time for the average to show up

Question 11

How does momentum work?

Answer 11

updates are calculated using an exponentially weighted average of the gradients that were calculated for previous batches

Question 12

What is the difference between standard and nesters momentum

Answer 12

With Nesterov momentum you calculate the gradient assuming you keep on moving in the direction you were moving instead of where you are.

Question 13

Name three algorithms with adaptive learning rates

Answer 13

• adagrad
• rmsprop
• Adam

Question 14

How does adagrad work?

Answer 14

Adagrad uses a different learning rate for each parameter 𝚹i.

The new theta 𝚹 =
the previous theta 𝚹 - (the learning rate / a small number to avoid devision by 0 + square root of the gradient for each parameter r) x the gradient

r gets bigger over time and so learning rate decreases

Question 15

Name an advantage and an disadvantage

Answer 15

+ It avoids the need for tuning of the learning rate

• The sum of squared gradients for some parameters may cause fast decrease in the learning rate and difficult learning
• Sum of squared gradient needs to be stored in memory

Question 16

What is the goal of RMSprop

Answer 16

The goal is to improve on the monotonically decreasing learning rate of Adagrad

Question 17

What is the difference between Adagrad and RMSprop?

Answer 17

In adagrad you use the squared root of the gradient, in RMSprop you use the squared root of the exponentially weighted average of the gradient.

Question 18

What does Adam stand for?

Answer 18

Adaptive Moment Estimation

Question 19

How does Adam work?

Answer 19

It uses exponentially weighted average of gradient and exponentially weighted average of squared average to calculate update. A corrections is introduced to avoid bias towards zero at the beginning of training.

Question 20

Explain adadelta, adamax and nadam.

Answer 20

Adadelta: Extends Adagrad

Adamax: Extends Adam

Nadam: Combines Adam and Nesterov

Question 21

What is an advantage of batch normalisation?

Answer 21

The influence of each input neuron is more equal.
Can avoid getting neurons to die out.
Can speed up learning

Question 22

How does batch normalisation work?

Answer 22

you take the activation of each neuron and subtract the average over the activation of the whole batch. Then you divide this by the square root of the squared SD + the epsilon the avoid division by 0. This is the Z Norm.
Next you multiply this with a parameter gamma and you add beta, both values are learned. This is done because if you have a lot of values around zero with sd =1, you can get stuck in the linear activation part of the activation function (s-hape).

In practice, during training the mean and standard deviation are calculated from training batches and then applied to the test set.

Question 23

Explain bias

Answer 23

Bias comes from assumptions made by a model to learn more easily.
● Low Bias: Less assumptions about the target function. (Too few parameters)
● High-Bias: More assumptions about the target function. (Too many parameters)

Question 24

Explain variance

Answer 24

Variance relates to how a estimated function would change if different training data is used
● Low Variance: Small changes with different training data. (Not memorizing noise)
● High Variance: Large changes with different training dataset. (Memorizing noise)

Question 25

Define regularisation

Answer 25

Changes in the learning of a model to make it satisfy certain constraints or preferences.

Question 26

Name example of regularization strategies

Answer 26

● Constraints on weights

● Additional terms on the objective (Loss) function

Question 27

How does regularisation work?

Answer 27

A paramater norm penalty Ω is added to the objective function.

The objective function is normally the loss function.
Normally only the weight are regularised and not the biases.

Question 28

what is l2 regularisation also known as?

Answer 28

weight decay

Question 29

How does L2 regularisation work?

Answer 29

basically you add the sum of squared parameters to the loss function this keeps the weights small

Question 30

How does L1 regularisation work?

Answer 30

basically you add the absolute value of parameters to the loss function, this keeps the weights small

Can lead to sparsity in weight parameters

Question 31

What is the goal of early stopping?

Answer 31

Keeping track of training and validation error to get get the best performing model.

The best model is to be defined in terms of the performance metrics.

Question 32

How does dropout work?

Answer 32

It randomly remove units for each training iteration.

Dropout trains an ensemble consisting of all subnetworks that can be constructed by removing non-output units from an underlying base network.

Question 33

What are the advantages of using dropout?

Answer 33

Dropout encourages modular representations which work well in the absence of some parts of the network (Hinton et al 2012).

One advantage of dropout is that it is very computationally cheap

Question 34

How is dropout implemented during training?

Answer 34

Sample a binary variable from a Bernoulli distribution with probability phidden -> Multiply unit activation with the sampled variable

Question 35

How is dropout implemented during testing?

Answer 35

● Maintain expected total input to a given unit at test time similar to the expected total input to that unit at train time (when some units were missing).
● Note: the alternative is to do inverse-scaling during training, and no scaling for testing.

The point is that the network gets the same size expected in training and testing.

Question 36

In what classification problem is data augmentation used the most?

Answer 36

object recognition

Question 37

How can you apply data augmentation in object recognition / images?

Answer 37

● translation
● crop
● scaling
● rotation
● flipping
● adding noise

Question 38

what is the vanishing and exploding gradients problem

Answer 38

In the network you keep multiplying weights and adding them up. So your output and loss function and gradient can get very big., which makes the updates in back-propagation very big –> exploding gradient.

If you have very small weights you can get really small activations. So you don’t learn anything. –> vanishing gradients

Question 39

How can you solve the vanishing and exploding gradients problem?

Answer 39

better initialisation of the parameters

A good practice to initialize weights is by drawing values from a normal distribution with zero mean and a specific variance.
The most used methods are:
● Xavier
● He

Question 40

How does the Xavier method work?

Answer 40

The variance is given by:
1 / fan_in
Where fan_in corresponds to the number of incoming neurons.

Question 41

How does the He method work?

Answer 41

The variance is given by:
2 / fan_in
Where fan_in corresponds to the number of incoming neurons.

Question 42

When should you use Xavier and when should you use He?

Answer 42

Xavier works better with Sigmoid or Tanh activation functions.

He works better with ReLu activation function