Lecture 15 - Neural networks: backpropagation Flashcards
What is forward propagation?
The process of computing outputs by passing inputs through the network layer by layer.
What are the components of a simple neural network?
Input layer: Takes raw data.
Hidden layers: Perform computations and transformations.
Output layer: Produces predictions.
What is backpropagation?
An algorithm to compute gradients of the loss function with respect to weights efficiently, layer by layer, using the chain rule.
Why is the chain rule important in backpropagation?
It allows the gradient of a composite function to be computed by breaking it into smaller, manageable derivatives.
Name and describe common activation functions.
Sigmoid: Maps inputs to (0, 1); used in binary outputs.
ReLU: 𝑔(𝑧)=max(0,𝑧); avoids vanishing gradients.
Softmax: Converts logits to probabilities for multi-class classification.
What is a computational graph?
A structured representation of computations where:
Nodes: Variables/operations.
Edges: Flow of computations.
What is the goal of gradient-based learning?
To minimize the cost function 𝐽 using methods like gradient descent.
What are the steps of backpropagation?
Forward Pass: Compute predictions.
Backward Pass: Calculate gradients.
Update Weights: Adjust weights using gradients.
How does dynamic programming optimize backpropagation?
By caching intermediate gradients to avoid redundant calculations.
How can overfitting in neural networks be prevented?
Dropout: Randomly deactivate units during training.
L2 Regularization: Penalize large weights.
Early Stopping: Halt training when validation loss stops improving.
