DL-02 - Improving DNN

Question 1

DL-02 - Improving DNN

What are 3 commonly used DL approaches to avoid overfitting? (3)

Answer 1

• regularization
• dropout
• early stopping

Question 2

DL-02 - Improving DNN

What are two common regularization techniques?

Answer 2

• L1
• L2

Question 3

DL-02 - Improving DNN

What is another name for L1 regularization?

Answer 3

Lasso

Question 4

DL-02 - Improving DNN

What is another name for Lasso regularization?

Answer 4

L1

Question 5

DL-02 - Improving DNN

What is another name for L2 regularization?

Answer 5

Ridge

Question 6

DL-02 - Improving DNN

What is another name for Ridge regularization?

Answer 6

L2

Question 7

DL-02 - Improving DNN

What are the basic types of learning rate decay? (5)

Answer 7

• Common method
• Exponential
• Epoch number based
• Mini-batch number based
• Discrete staircase

Question 8

DL-02 - Improving DNN

What is the formula for “common method” learning rate decay?

Answer 8

(See image)

Question 9

DL-02 - Improving DNN

What is the name for this method of learning rate decay? (See image)

Answer 9

“Common method” learning rate decay

Question 10

DL-02 - Improving DNN

What is the formula for exponential learning rate decay?

Answer 10

(See image)

Question 11

DL-02 - Improving DNN

What is the name for this method of learning rate decay? (See image)

Answer 11

Exponential learning rate decay

Question 12

DL-02 - Improving DNN

What is the formula for “epoch number based” learning rate decay?

Answer 12

(See image)

Question 13

DL-02 - Improving DNN

What is the name for this method of learning rate decay? (See image)

Answer 13

“epoch number based” learning rate decay

Question 14

DL-02 - Improving DNN

What is the formula for “mini-batch number based” learning rate decay?

Answer 14

(See image; should say “mini batch number”)

Question 15

DL-02 - Improving DNN

What is the name for this method of learning rate decay? (See image; should say “mini batch number”)

Answer 15

mini-batch number based

Question 16

DL-02 - Improving DNN

What is the name of this method of learning rate decay? (See image)

Answer 16

Discrete staircase.

Question 17

DL-02 - Improving DNN

What does the learning rate graph look like with a “discrete staircase” approach?

Answer 17

(See image)

Question 18

DL-02 - Improving DNN

What is the role of momentum in neural network training?

Answer 18

Momentum reduces oscillations and speeds up convergence for a smoother learning process.

Question 19

DL-02 - Improving DNN

How does AdaGrad help improve the performance of a neural network?

Answer 19

By adaptively scaling learning rates based on accumulated past gradients for each parameter, leading to faster convergence.

Question 20

DL-02 - Improving DNN

Which optimization algorithm is AdaDelta built on?

Answer 20

AdaGrad

Question 21

DL-02 - Improving DNN

What’s the difference between AdaDelta and AdaGrad?

Answer 21

AdaDelta improves AdaGrad by addressing its diminishing learning rates issue through using a moving average of squared gradients instead of historical accumulated gradients.

Question 22

DL-02 - Improving DNN

How does RMSprop work?

Answer 22

RMSprop works by adapting the learning rate for each weight parameter using a running average of the magnitude of recent gradients.

Question 23

DL-02 - Improving DNN

How does the Adam optimizer work?

Answer 23

The Adam optimizer works by adaptively adjusting learning rates for each parameter using both moment estimates and exponentially-averaged past gradients.

Question 24

DL-02 - Improving DNN

What are some benefits of using ADAM over other optimizers? (AESN)

Answer 24

• Adaptive learning rates
• efficient computation
• suitable for sparse data
• reduced noise in parameter updates

Question 25

DL-02 - Improving DNN

Which optimizer performs the best on average?

Answer 25

ADAM.

Question 26

DL-02 - Improving DNN

What is a requirement for using batch normalization?

Answer 26

Decently large mini-batches. Small batches make it unstable.

Question 27

DL-02 - Improving DNN

How does batch normalization act as a regularizer?

Answer 27

It adds noise during training.

Question 28

DL-02 - Improving DNN

How does batch and layer normalization differ?

Answer 28

• BN normalizes batches, e.g. same params for a whole image.
• LN normalizes layers/features separately, but also inside the batch.

Question 29

DL-02 - Improving DNN

Is batch normalization stable for small mini-batch sizes?

Answer 29

No, need large batch sizes.

Question 30

DL-02 - Improving DNN

Is layer normalization stable for small mini-batch sizes?

Answer 30

Yes, it’s not dependent on batch size.

Question 31

DL-02 - Improving DNN

What are the 3 types of hyper parameter tuning approaches? (3)

Answer 31

• Manual
• Brute force (Grid search, random search etc.)
• Meta model (Machine learning, e.g. Optuna)

Question 32

DL-02 - Improving DNN

What is HPO short for?

Answer 32

Hyperparameter optimization

Question 33

DL-02 - Improving DNN

What is a surrogate model in therms of HPO?

Answer 33

A model trained on the hyper parameters, where the model output is model quality.

Question 34

DL-02 - Improving DNN

What is a requirement for choosing what model to use for a surrogate model?

Answer 34

No way to know gradient -> model should use gradient free optimization.

Question 35

DL-02 - Improving DNN

When should you retune hyperparameters?

Answer 35

Occasionally/regularly, especially with a change in the data or problem to solve.

Question 36

DL-02 - Improving DNN

What is NAG short for?

Answer 36

Nesterov accelerated gradient

Question 37

DL-02 - Improving DNN

What is the purpose of Nesterov momentum in neural network training?

Answer 37

Nesterov momentum accelerates training by computing future position gradients, reducing oscillations and improving convergence.

Question 38

DL-02 - Improving DNN

What is RMSProp short for?

Answer 38

Root Mean Square Propagation