! S9: Gradient Descent

Question 1

Gradient Descent - Definition

Answer 1

iterative optimization algorithm used to minimize a cost function to find the optimal values of the parameters (weights) in a machine learning model

Question 2

Gradient Descent - Steps

Answer 2

1. Define a cost function (difference btw predicted & actual values)
2. Calculate the partial derivatives of cost function with respect to each weight (random initalized first)
3. Update weights: if derivative >0 -> decrease weight
4. Repeat until reach minimum of the cost function

Question 3

Gradient Descent - Step Size

Answer 3

• how much update weights in each iteration
• step size = slope * learning rate
• steps gradually get smaller as parameters approach minimum

Question 4

Gradient Descent - Con

Answer 4

• all features must have similar scale
• high computational cost O(ndt) (10.000 dp * 10 features * 1.000 iterations)
• might only find local minimum (if non-convex)

Question 5

Gradient

Answer 5

= derivative of a function that has > 1 input variable

Question 6

Stochastic GD

Answer 6

= solution to reduce computational cost: randomly picks one dp at each iteration to reduce computations -> n = smaller

Question 7

Batch & Mini Batch GD

Answer 7

• solution to reduce computational cost: randomly picks minibatches at each iteration & calculates GD
• improve performance by reducing variance (<-> SGD)

Question 8

Stochastic GD - pro

Answer 8

• efficient -> indepentent of n -> O(dt)
• easy implementation
• avoidance of local minima due to noisy update of weights
• each training step is faster (but more variance)

Question 9

Stochastic GD - Con

Answer 9

• need hyperparamets (regularization parameter, number of iterations)
• sensitive to feature scaling
• Gradient of random example might point in wrong direction -> solution: ok if most gradients points in right direction
• noisy updates & high variance (optimization less stable)
• slow convergence: might nedd more iterations to find minimum
• sensitive to learning rate: to low -> can overshoot minimum

Question 10

Regularization

Answer 10

• methods to prevent overfitting
• goal: low bias (training erro) & low variance (test error)

Question 11

Methods to prevent overfitting

Answer 11

1. Methods with minimizing loss function + penalty (during training) Regularization
2. Early Stopping
3. Dropout
4. BatchNorm

Question 12

Regularization - Methods with minimizing loss function + penalty

Answer 12

• techniques to select features by penalizing small weights if feature not necessary for model
• add penalty to loss function to select features
• goal: avoid under- & overfitting

Question 13

Regularization - Methods with minimizing loss function + penalty - Techniques

Answer 13

1. L0 regularization
2. L1 regularization (lasso)
3. L2 Regularization (Ridge)
4. Elastic Net (L1 + L2)

Question 14

L0 regularization

Answer 14

penalty is one number (lambda if w not 0)

Question 15

L1 Regularization (Lasso)

Answer 15

• penalty added to loss function: sum of “absolute value of magnitude” of coefficients (penalty on L1-norm of “w”)
• Loss function + lambda sum IwI

Question 16

L2 Regularization (Ridge)

Answer 16

• penalty added to loss function: sum of “squared magnitude” of coefficients (penalty on L2-norm of “w”)
• Loss function + lambda sum IwI^2
• Lambda = hyperparameter of how strong to penalize

Question 17

L2 & L1 Regularization - Similarity

Answer 17

• decrease variance: lower test error

Question 18

L2 & L1 Regularization - Difference

Answer 18

• L1: encourages w to be exactly 0 (if weight = 0 penalizes inclusion of feature in model), solution not unique
• L2: all w tend to be non-zero, unique solution

Question 19

Early Stopping

Answer 19

= technique to prevent overfitting by monitoring models performance on validation set during training & stopp at lowest validation error (training error can go further down)

Question 20

Early Stopping - Method

Answer 20

• Algorithm learns in “epochs”: takes whole data multiple times during training, makes predictions, compare to actual labels & updates weights & biases according to minimizing loss
• Monitor validation error as run Stochastic Gradient
• Stop algorithm if validation error reaches minimum / increases again (= overfitting)

Question 21

Dropout

Answer 21

= technique to prevent overfitting by randomly dropping out (i.e., setting to zero) a proportion of the output features or activations of a layer during training – for deep neural networks

Question 22

Dropout - Method

Answer 22

• At each training step: each input neuron has probability p (dropout rate) of being dropped
• Evaluate training loss: Overfits -> increase, underfits -> decrease
• After training: use all neurons
• Compensation for dropout: 1 Multiply each input connection weight by (1-p), 2. Divide each neurons output by p

Question 23

Dropout - Time-Effort-trade-off

Answer 23

well-tuned dropout -> slows convergence but better model

Question 24

Batch Norm

Answer 24

technique used to mitigate the negative effects of the internal covariate shift by stabilizing distribution of inputs (over a minibatch) to each layer during training

Question 25

Batch Norm - Importance

Answer 25

• In Deep neural networks: each layer receives input -> gives output to next layer
• During training: values of inputs to each layer can change
• change in distribution of input values = internal covariate shift (= statistical properties of inputs to layer, e.g. mean & variance, differ as network learns)
• challenging for subsequent layers to learn effectively because they need to continuously adapt to changing distributions

Question 26

Batch Norm - Method

Answer 26

• opeates on mini-batch of inputs -> for each mini-batch mean & variance of activations across batch computed
• at each hidden layer:
• normalize & zero-center input
• Scale & shift results
• Pass through Activation function
• Fed into next layers

Question 27

Batch Norm - Pro

Answer 27

• Regularization effect (as adds some noise to mini-batch statistics -> prevents overfitting)
• Reduced sensitiveity to initalization
• no standardization in training set required if batch normalization layer at first layer
• better performance
• convergence faster

Question 28

Batch Norm - Con

Answer 28

• Adds runtime penalty to NN (training slower because each epoch takes time for BN)
• Tricky to use in RNNs (<-> easier in e.g. CNN)

Question 29

Gradient Clipping

Answer 29

• often used in RNN
• solution to exploiding gradient probelm
• sets threshold on gradients: if to large -> gradients scaled down or clipped
• stabilize training process & prevents extreme updates to model’s parameters