Regularization & Optimizers

Question 1

Why does L2 regularization shrink weights instead of setting them to zero?

Answer 1

L2 regularization penalizes the squared magnitude of weights, shrinking them proportionally without forcing them to zero.

Question 2

How does L1 regularization change the gradient of the loss function?

Answer 2

L1 regularization adds the term λ * sign(w) to the gradient, pushing weights toward zero.

Question 3

What is Dropout?

Answer 3

Dropout is a regularization technique that randomly sets a fraction of neurons to zero during training, reducing reliance on specific neurons and improving generalization.

Question 4

What is Data Augmentation?

Answer 4

Data Augmentation involves creating variations of training data (e.g., rotations, flips, noise) to improve model generalization and robustness.

Question 5

What is Early Stopping?

Answer 5

Early Stopping monitors validation performance during training and stops training when the performance no longer improves, preventing overfitting.

Question 6

What is Batch Normalization?

Answer 6

Batch Normalization normalizes layer inputs within a mini-batch, stabilizing training and acting as a form of regularization.

Question 7

What is Weight Decay?

Answer 7

Weight Decay applies a sort of L2 regularization during training, penalizing large weights to improve generalization.

Question 8

What is Noise Injection?

Answer 8

Noise Injection adds noise to inputs, weights, or activations during training to make the model more robust and prevent overfitting.

Question 9

What is SGD (Stochastic Gradient Descent)?

Answer 9

SGD updates model weights based on a small random subset of data (a mini-batch), improving training efficiency and convergence.

Question 10

What is Momentum in optimization?

Answer 10

Momentum is an extension of SGD that accelerates training by using a fraction of the previous weight update in the current update.

Question 11

What is RMSprop?

Answer 11

RMSprop adapts the learning rate for each parameter by dividing the gradient by a moving average of recent squared gradients, peanalising bigger gradients at points.

Question 12

What is Adam Optimizer?

Answer 12

Adam combines the advantages of Momentum and RMSprop, using adaptive learning rates and momentum for faster and more stable convergence.

Question 13

How do L1 and L2 regularization differ in their impact on sparsity?

Answer 13

L1 regularization creates sparsity by driving some weights to zero, while L2 shrinks weights without eliminating them.

Question 14

When would you prefer L1 regularization over L2 regularization?

Answer 14

L1 is preferred for sparse data or when feature selection is needed.

Question 15

Why is L0 regularization difficult to optimize?

Answer 15

L0 regularization is non-differentiable and discontinuous, making gradient-based optimization infeasible.

Question 16

What is the primary difference between SGD and Adam?

Answer 16

Adam uses momentum and adaptive learning rates, while SGD applies a fixed learning rate without momentum.

Question 17

What are the key benefits of momentum-based optimizers over SGD?

Answer 17

Momentum reduces oscillations and speeds up convergence by smoothing updates.

Question 18

When is a learning rate scheduler useful?

Answer 18

A learning rate scheduler is useful when training plateaus or oscillates to stabilize convergence.

Question 19

How do Adam and RMSprop differ in adjusting learning rates, and why is Adam often preferred?

Answer 19

Adam builds on this by adding momentum ( 𝑚 𝑡 ) for gradient smoothing and bias correction for both moments, making it more robust across diverse tasks.