lecture 6 - DNNs

Question 1

How do shallow networks differ from deep networks?

Answer 1

Shallow networks have fewer hidden layers (1 or 2), while deep networks have many hidden layers (e.g., 5 or more).

Question 2

Why are deep neural networks considered a separate research area?

Answer 2

Deep networks come with additional complexities in learning. Traditional gradient descent doesn’t work well for them, requiring specialized techniques.

Question 3

Why are deep networks necessary for tasks like image recognition and language processing?

Answer 3

The additional layers and nodes in deep networks enable the creation of more complex features, which are essential for these tasks.

Question 4

Why use deep neural networks instead of shallow networks that are already universal approximators?

Answer 4

Deep neural networks can compute many functions with fewer layers, whereas shallow networks require exponentially more hidden units to achieve the same level of approximation.

Question 5

How do deep and shallow networks compare in terms of required resources for complex problems?

Answer 5

1. Deep networks require only a logarithmic number of neurons O(log n) for certain problems.
2. Shallow networks require exponentially many neurons
   O(2^n) for the same problems.

• Deep networks exhibit logarithmic growth, whereas shallow networks exhibit exponential growth in the number of required neurons.

Question 6

What is an example of a problem where deep networks outperform shallow networks in terms of efficiency?

Answer 6

For problems like x_1 XOR x_2 XOR x_3 XOR x_4, deep networks can solve the problem with far fewer neurons than shallow networks

Question 7

What is forward propagation in DNNs?

Answer 7

Forward propagation is the process of passing input data through the layers of a neural network to compute the final output.

Question 8

What are the two main operations performed by each layer in a neural network during forward propagation?

Answer 8

1. linear transformation: Inputs from the previous layer are multiplied by a weight matrix and added to a bias vector, resulting in the pre-activation values.
2. non-linear activation: The pre-activation values are passed through an activation function to compute the final output layer

Question 9

Why is a non-linear activation function important in forward propagation?

Answer 9

The non-linear activation function introduces non-linearity, enabling the neural network to learn and model complex patterns.

Question 10

What is backward propagation in DNNs?

Answer 10

Backward propagation is the process used to calculate the gradient of the loss function with respect to the weights and biases in the network. The gradients are used to update the parameters via optimization (e.g., gradient descent).

Question 11

What is the starting point for backward propagation?

Answer 11

The starting point is the derivative of the loss with respect to the output, da^[l], in the final layer L.

Question 12

What are the key steps in computing the gradient flow for each layer during backward propagation?

Answer 12

1. compute the gradient of the pre-activation values (dz^[l])
2. compute the gradient of the weights (dW^[l])
3. compute the gradient of the biases (db^[l])
4. backpropagate the error to the previous layer (da^[l-1])

Question 13

What are hyperparameters in a neural network?

Answer 13

Hyperparameters are parameters set before training a neural network, influencing the learning process and performance of the model.

Question 14

How does the number of layers affect a neural network?

Answer 14

• The number of layers refers to the depth of the network.
• More layers allow the network to learn complex hierarchical features but increase computational complexity and the risk of overfitting.

Question 15

What does the number of units per layer determine in a neural network?

Answer 15

The number of units per layer determines the width of the network. More units increase the model’s capacity but may also lead to overfitting.

Question 16

Why is learning rate an important hyperparameter in gradient descent?

Answer 16

• The learning rate controls the step size during gradient descent.
• A high learning rate might overshoot the minimum, while a low learning rate can make training slow.

Question 17

What role do activation functions play in neural networks?

Answer 17

• Activation functions define the transformation applied to the input at each layer.
• Common choices include ReLU, Sigmoid, and Tanh.

Question 18

Important hyperparameters

Answer 18

1. layers
2. units
3. learning rate
4. activation functions
5. batch size
6. weight initialization strategies
7. dropout rate
8. optimizers

Question 19

How should datasets be divided to evaluate and optimize hyperparameters?

Answer 19

1. Training set: Used to train the model.
2. Development set: Used to tune the hyperparameters.
3. Test set: Used to evaluate the performance of the algorithm.

Question 20

What are bias and variance in the context of neural networks?

Answer 20

1. Bias: Error due to overly simplistic assumptions in the model. High bias indicates underfitting.
2. Variance: Error due to high sensitivity to small fluctuations in the training data. High variance indicates overfitting.

Question 21

Why should you address bias before variance in a model?

Answer 21

• High bias indicates that the model poorly approximates the data regardless of variance.
• Then, decrease the variance
• Since decreasing variance can increase bias, it’s crucial to then check the bias again

Question 22

What are the characteristics of a model with high variance?

Answer 22

• A high-variance model performs well on the training set but poorly on the validation/dev set.
• Example: Train set error = 1%, Dev set error = 11%.

Question 23

How can you reduce high variance in a neural network?

Answer 23

1. Increase the size of the training data.
2. Apply regularization.
3. Adjust the network architecture (e.g., add more layers or units).

Question 24

What are the characteristics of a model with high bias?

Answer 24

• A high-bias model performs poorly on both the training and validation/dev sets.
• Example: Train set error = 15%, Dev set error = 16%.

Question 25

How can you reduce high bias in a neural network?

Answer 25

1. Use a larger network.
2. Train for a longer duration.
3. Adjust the architecture (e.g., add more layers or units).

Question 26

What are the characteristics of a model with both high bias and high variance?

Answer 26

• The model has large errors on both the training set and an even larger error on the dev set, indicating that it neither fits the training data well nor generalizes.
• Example: Train set error = 15%, Dev set error = 30%.

Question 27

How can you address high bias and high variance?

Answer 27

Address the bias first (by increasing model capacity or training longer) and then reduce variance (by applying regularization or using more data).

Question 28

What are the characteristics of a model with low bias and low variance?

Answer 28

• This is the ideal case where both training and dev errors are low, with the dev set error slightly higher than the training set error.
• Example: Train set error = 0.5%, Dev set error = 1%.

Question 29

high variance plot

Answer 29

circles exactly around all the targets

Question 30

high bias plot

Answer 30

rigid straight/slanted line that captures most of the points

Question 31

high bias high variance plot

Answer 31

straight line that makes an exception for outlying point

Question 32

low bias low variance plot

Answer 32

circles around most of the target points but leaves the outlying point out.

Question 33

What is dropout regularization in neural networks?

Answer 33

Dropout is a regularization technique used to prevent overfitting by randomly dropping out a proportion of neurons during the training process. The dropped-out neurons do not participate in forward and backward propagation.

Question 34

How does dropout regularization help in preventing overfitting?

Answer 34

By randomly turning off neurons, dropout forces the network to distribute the learned weights across different neurons, preventing any single neuron from becoming too dominant and improving the model’s generalization ability.

Question 35

Why is it impractical to use one regularization term for both layers in deeper neural networks?

Answer 35

As the number of layers increases, using a single regularization term becomes impractical because it adds too many hyperparameters to tune. Dropout addresses this issue without adding extra hyperparameters.

Question 36

What do L2 regularization and dropout target?

Answer 36

1. L2 Regularization: Prevents overly large weights.
2. Dropout: Prevents reliance on specific neurons.

Question 37

What is early stopping in neural network training?

Answer 37

• Alternative to regularization
• Early stopping involves halting the training process at the point where the validation error is smallest, preventing overfitting by not allowing the model to train further once it starts overfitting.

Question 38

How does data augmentation help in reducing overfitting?

Answer 38

Data augmentation artificially increases the size and diversity of the training dataset by applying transformations (e.g., rotations, translations) to the data, making the model more robust to variations.

Question 39

What is the key benefit of data augmentation?

Answer 39

It helps models learn to ignore irrelevant variations, such as minor shifts or distortions, and focus on the important features that define the object or class.

Question 40

Why is input normalization important for optimization using gradient descent?

Answer 40

Without normalization, input features with varying scales can lead to elongated cost function contours, making gradient descent inefficient as it must zig-zag through narrow valleys to converge.

Question 41

How does normalizing the inputs affect the cost function contours?

Answer 41

Normalizing inputs (scaling to have a mean of 0 and variance of 1) results in circular contours, allowing gradient descent to converge faster by following a more direct optimization path.

Question 42

What are batch normalized layers, and how do they help?

Answer 42

Batch normalized layers normalize activations after a few forward propagation layers, then propagate forward again. This helps maintain learnability and improves training stability and speed.

Question 43

What is the key takeaway about normalizing inputs for deep learning models?

Answer 43

Always normalize input features to improve training stability and speed in deep learning models.

Question 44

What are vanishing and exploding gradients, and why are they problematic in deep learning?

Answer 44

• In very deep networks, the output can be expressed as a product of many weight matrices.
• If the largest eigenvalue of
  W is greater than 1, gradients explode
• If the largest eigenvalue of
  W is less than 1, gradients vanish.
• Both scenarios cause instability during training.

Question 45

What happens during vanishing gradients, and what is the effect on training?

Answer 45

• Gradients become very small (approach 0) as they backpropagate through deep layers, especially with activation functions like Sigmoid or Tanh.
• This causes the weights in earlier layers to update very slowly, leading to poor learning.

Question 46

How can vanishing gradients be mitigated?

Answer 46

One solution is to pass information of the gradient to the layer after the next layer (skip layer connection), helping maintain gradient flow.

Question 47

What happens during exploding gradients, and what is the effect on training?

Answer 47

• Gradients grow excessively large in deeper layers, causing instability in learning and
• Leads to large oscillations in gradient descent or numerical overflow.

Question 48

How can exploding gradients be mitigated?

Answer 48

One solution is gradient clipping, which limits the maximum value of the gradient to prevent large jumps in gradient descent.

Question 49

What is mini-batch gradient descent?

Answer 49

• Mini-batch gradient descent is an optimization algorithm that divides the training dataset into smaller subsets called batches.
• The algorithm computes gradients based on a batch rather than the entire dataset or a single sample.

Question 50

What is the advantage of using mini-batch gradient descent over batch gradient descent?

Answer 50

Mini-batch gradient descent allows faster computation by leveraging vectorization and reduces memory requirements compared to batch gradient descent.

Question 51

How does forward propagation work in mini-batch gradient descent?

Answer 51

In mini-batch gradient descent, forward propagation is performed on a set of data (a mini-batch), the error is measured, and the derivative of the error is used in backpropagation.

Question 52

How does the cost decrease differently in batch gradient descent versus mini-batch gradient descent?

Answer 52

1. Batch Gradient Descent: The cost decreases steadily with iterations.
2. Mini-batch Gradient Descent: The cost converges to the same place as batch gradient descent but oscillates around it.

Question 53

What is batch gradient descent?

Answer 53

Batch gradient descent uses the entire training dataset to compute the gradient for each iteration, resulting in smooth convergence but being computationally expensive for large datasets.

Question 54

How is batch gradient descent represented in visualizations?

Answer 54

It is represented in blue, showing a smooth path toward the optimum.

Question 55

What is stochastic gradient descent (SGD)?

Answer 55

SGD updates the weights after computing the gradient for a single data point, resulting in much faster updates but noisier and more erratic convergence, with a higher risk of overshooting the minimum.

Question 56

How is stochastic gradient descent represented in visualizations?

Answer 56

It is represented in purple, showing a zig-zag, erratic path.

Question 57

How does mini-batch gradient descent combine the benefits of batch and stochastic gradient descent?

Answer 57

Mini-batch gradient descent combines the faster updates of SGD with the smoother trajectory of batch gradient descent, circling around the optimal value.

Question 58

How is mini-batch gradient descent represented in visualizations?

Answer 58

It is represented in green, showing a less chaotic path than SGD.

Question 59

What is gradient descent with momentum, and why is it used?

Answer 59

• Gradient descent with momentum adds a “memory” to the updates by incorporating the previous update direction.
• This helps dampen oscillations in directions where updates repeatedly reverse (e.g., ellipsoids), allowing the optimization to approach the minimum more smoothly.

Question 60

How are weights and biases updated in gradient descent with momentum?

Answer 60

• W = W − α v_dw
  ​
• b = b − α v_db

​- where α is the learning rate.

Question 61

How are the velocity terms v_dw and v_db computed in gradient descent with momentum?

Answer 61

• v_dw = βv_dw +(1−β)dW
• v_db = βv_db +(1−β)db
• where β is the damping factor
• dW and db are the current gradients.
• v_dw and v_db (in equation) are the old gradients
• this is dynamically averaging the previous gradient estimates and adding this to gradient descent to direct the weight updates

Question 62

What are the hyperparameters in gradient descent with momentum?

Answer 62

1. Learning rate α
2. the damping factor β (commonly set to 0.9), which determines how much of the past momentum is retained.

Question 63

What is a running average in the context of optimization?

Answer 63

• A running average is an example of exponential smoothing, where recent values are given slightly more importance, helping to smooth out noisy updates.

Question 64

How is a running average conceptually similar to the momentum update in optimization algorithms?

Answer 64

Both combine past gradients with the current one to smooth out noisy updates, resulting in more stable and consistent progress toward the minimum.

Question 65

How does momentum affect gradient descent?

Answer 65

Momentum averages all the gradient estimates, adding it to the gradient descent process, leading to a more stable pattern with reduced oscillations and overshooting.

Question 66

How does momentum help optimization in terms of local minima?

Answer 66

Momentum speeds up the optimization in directions without oscillations, enabling it to escape shallow local minima and reach the global minimum more efficiently.

Question 67

What issue does RMSProp address that momentum does not?

Answer 67

• RMSProp addresses the issue of not considering the variance
• Does this by keeping a moving average of the squared gradients, scaling the learning rate dynamically for each parameter.

Question 68

How does RMSProp stabilize training?

Answer 68

By adjusting the step size for each parameter dynamically based on the magnitude of past gradients, ensuring steady progress in high curvature or noisy regions of the loss landscape.

Question 69

What is the role of s_dw and s_db in RMSProp?

Answer 69

They represent the moving averages of the squared gradients for weights W and biases b, used to normalize the update step.

Question 70

How are the weight and bias updates computed in RMSProp?

Answer 70

• W = W - α (dW/ sqrt(s_dw))
• b = b - α (db/ sqrt(s_db))

Question 71

What is the recursive formula for the squared gradient moving average in RMSProp?

Answer 71

s_T = (1-β) sum(β^{t-i} * g^2_i)

Question 72

How does RMSProp adjust the step size based on uncertainty?

Answer 72

• For high uncertainty (large or fluctuating gradients), the step size is small as α is diminished by the term.
• For high certainty (consistent gradients), the step size is larger as α remains unaffected.

Question 73

What is Adam optimization a combination of?

Answer 73

1. momentum (which smooths gradient updates using first-order moments)
2. RMSProp (which adapts learning rates using second-order moments).

Question 74

What is the key problem with RMSProp during early training, and how does Adam address it?

Answer 74

• The key problem with RMSProp during early training is that the moving averages of past gradients start close to zero and only gradually stabilize as more data points are processed.
• This contraction toward zero makes the updates too conservative, resulting in very small step sizes that can slow down learning significantly.
• Adam addresses this by applying bias correction, which adjusts the moving averages of the gradients and squared gradients to account for their initial underestimation.
• This ensures that the step sizes reflect the true gradient magnitude early in training, allowing for more effective exploration of the loss surface.

Question 75

How does Adam optimization combine the strengths of momentum and RMSProp?

Answer 75

momentum

• Computes moving averages of the gradients (v_dw and v_db) using an exponential decay factor (β_1, typically set to 0.9).
• This smooths the updates and helps dampen oscillations, ensuring more stable convergence.

RMSprop

• Computes the moving averages of the squared gradients (s_dw and s_db) using an exponential decay factor (β_2, typically set to 0.999)
• This helps adapt the learning rate dynamically by normalizing updates based on the magnitude of recent gradients, preventing overly large updates in steep regions or small updates in flat regions.

Question 76

What are the hyperparameters of Adam, and what are their typical values?

Answer 76

1. α: The learning rate, typically set to 0.001.
2. β_1: Decay rate for the moving average of gradients, typically set to 0.9.
3. β_2: Decay rate for the moving average of squared gradients, typically set to 0.999.
4. ϵ: A small constant for numerical stability, typically set to 10^-8

Question 77

What are the key advantages of Adam optimization?

Answer 77

1. Adaptive learning rates: By combining momentum and RMSProp, Adam adapts the learning rate for each parameter dynamically, leading to more efficient optimization.
2. Bias correction: Ensures accurate step sizes even during early training when moving averages are still stabilizing.
3. Fast convergence: Due to its bias correction and adaptive learning rates, Adam often converges faster than other optimizers.
4. Robustness: Works well for a wide range of deep learning problems and requires little hyperparameter tuning compared to other algorithms.

Question 78

What is the main behavior of Gradient Descent?

Answer 78

• Takes small, consistent steps downhill.
• follows a straightforward but slow trajectory. It struggles in the narrower valley, requiring multiple steps to align itself with the global minimum.

Question 79

What is the main challenge of Gradient Descent?

Answer 79

It can be slow in regions with shallow gradients and may oscillate in narrow valleys.

Question 80

How does Momentum improve upon Gradient Descent?

Answer 80

• It accelerates progress in consistent gradient directions by adding a “velocity” term and dampens oscillations.
• smoother strides and avoids oscillations. It converges more quickly into the global minimum by effectively handling the steep slopes.

Question 81

What is the advantage of using Momentum?

Answer 81

It helps navigate narrow valleys and speeds up convergence compared to plain Gradient Descent.

Question 82

How does RMSprop handle steep slopes and flat regions effectively?

Answer 82

By using an exponentially weighted moving average of squared gradients to adaptively scale the learning rate.

Question 83

What are the advantages of RMSprop?

Answer 83

It converges efficiently in both steep and flat regions, avoiding overly aggressive learning rate decay.

Question 84

How does Adam optimization combine the features of other methods?

Answer 84

It combines Momentum and RMSprop, using both a running average of gradients and their squared values with bias correction.

Question 85

What happens to all optimizers at saddle points?

Answer 85

• Adam avoids getting stuck at saddle points
• All other optimizers get stuck.

Question 86

Which optimizers can reach the global minimum with a slight increase in convexity?

Answer 86

Momentum and Adam.

Question 87

What is the primary goal of hyperparameter tuning?

Answer 87

The goal is to optimize hyperparameters to minimize a loss function f(x) by finding the best set of values that improve model performance.

Question 88

Why should you avoid using grid search for hyperparameter tuning?

Answer 88

Because it leads to exponential explosion in the number of evaluations, making it computationally expensive.

Question 89

What is a more efficient alternative to grid search for hyperparameter tuning?

Answer 89

Random search or Bayesian optimization is recommended as it explores the parameter space more efficiently.

Question 90

What is an effective strategy for hyperparameter tuning?

Answer 90

1. Go from coarse to fine.
2. Pick an appropriate scale for each hyperparameter.
3. Run the optimization in parallel.

Question 91

How does random search differ from grid search in hyperparameter tuning?

Answer 91

Bayesian optimization uses a probabilistic model (surrogate) to predict the performance of different hyperparameter values and focuses on areas with high expected improvement.

Question 92

What does Bayesian optimization use to model the unknown function?

Answer 92

It uses a surrogate model, often a Gaussian Process (GP), which provides predictions and uncertainty estimates for the function.

Question 93

What are the key components of a Gaussian Process used in Bayesian optimization?

Answer 93

The key components are the mean function (prediction) and the covariance function (measuring similarity between points).

Question 94

How does Bayesian optimization decide where to evaluate the next point in the hyperparameter space?

Answer 94

It uses a metric called Expected Improvement (EI) to choose the next point by balancing exploration and exploitation.

Question 95

What does the Expected Improvement (EI) metric measure in Bayesian optimization?

Answer 95

EI measures the expected gain from evaluating a new point, considering both the potential for improvement and uncertainty.

Question 96

How does the surrogate model evolve during the Bayesian optimization process?

Answer 96

As more evaluations are performed, the surrogate model becomes more accurate by interpolating the observed data points better.

Question 97

Why are uncertainty bounds important in Bayesian optimization?

Answer 97

They help determine areas with high uncertainty where further sampling could yield significant improvements.

Question 98

What is the key insight behind using few curves vs. many curves in Bayesian optimization?

Answer 98

Few curves indicate high uncertainty, while many curves show increasing confidence in specific regions as more evaluations are performed.

Question 99

What does a sharp peak in the histogram during Bayesian optimization indicate?

Answer 99

It indicates that the model is highly confident about the location of the best hyperparameter.

Question 100

What happens to uncertainty as you sample more points around a specific region?

Answer 100

Uncertainty decreases as the model gains more information about that region.

Question 101

In Bayesian optimization, why does sampling far from known points often lead to higher expected gain?

Answer 101

Sampling far from known points that are not the minimum explores new areas with high uncertainty, potentially leading to large improvements.

Question 102

What is the purpose of using a surrogate model in Bayesian optimization?

Answer 102

The surrogate model approximates the true loss function to guide the search for the best hyperparameters more efficiently.

Question 103

Why is parallel evaluation important in hyperparameter tuning?

Answer 103

Parallel evaluation speeds up the tuning process by testing multiple hyperparameter sets simultaneously.

Question 104

Interpretation of the combination of the EI function

Answer 104

• tells us how to use this distribution to find out a next point to sample
• takes the best (lowest) point i’ve seen so far and compare it to the point i want to sample and multiply it by the density that ive estimated
• if there is a large difference between what i measure, but the density shows a low likelihood of the minimum being at that location, you won’t use it (outputs a low value).
• if there is a large difference and the distribution shows a high likelihood, this function outputs a high value.