lecture 5: backpropagation

Question 1

single-layer perceptrons

Answer 1

• are limited to linearly separable problems
• we need to add layers to make universal function approximators

Question 2

how to find the weights of a multilayer perceptron

Answer 2

with backpropagation

Question 3

What is the main purpose of backpropagation?

Answer 3

Backpropagation calculates and propagates errors backward through the network to adjust weights, enabling the network to learn by minimizing error.

Question 4

backpropagation steps

Answer 4

1. forward sweep
2. compare predicted output to true output
3. compute the error term
4. update the weight from output to hidden layer
5. update the weights of deeper layers

Question 5

What is forward propagation in a neural network?

Answer 5

• passing input data 𝑥 through the network to compute the output 𝑦 via the intermediate hidden layer activations ℎ.

Question 6

How is information flow represented mathematically in forward propagation?

Answer 6

• the flow is x→h→y
• Hidden layer activations: h=W^{hx}x
• Output: y=W^{yh}h

Question 7

What is the first step in backpropagation after the forward sweep?

Answer 7

Compare the predicted output y to the target t to calculate the error in the output layer.

Question 8

How is the error δ for the output layer calculated?

Answer 8

• δ=g′(a_j)⋅(t_j−y_j)
• output error = derivative of the activation * difference between target and predicted output

Question 9

How are weights connected to the output layer updated in backpropagation?

Answer 9

• Δw_jk =−ϵ⋅δ⋅h_k
• learning rate * error term * input to the weight

Question 10

How does backpropagation work for the hidden layers?

Answer 10

• propagates the error from the output layer back to the hidden layers
• δ_i=g′(a_i)⋅δ_j

Question 11

What are the two key steps in backpropagation?

Answer 11

1. Compute the error term δ for each layer.
2. Update the weights using the error term and the learning rate.

Question 12

neural network architectures

Answer 12

1. recursive neural network
2. convolutional neural network

Question 13

RNNs

Answer 13

• time as a factor: memory, sequence analysis, temporal predictions, language
• feed their outputs back into themselves, time-step by time-step

Question 14

RNNs: unrolling

Answer 14

expanding the RNN over time steps, treating each time step as a layer in the network for backpropagation through time (BPTT)

Question 15

RNNs: feedback loop

Answer 15

lets RNNs maintain a record of history of past inputs, and integrate information over time

Question 16

RNNs: learning

Answer 16

• backpropagation through time (BPTT)
• backpropagation is extended over time to compute gradients for all time steps in the sequence.
• this allows RNNs to learn how earlier time steps influence later ones.

Question 17

RNNs: pros

Answer 17

• can learn all sorts of sequences, even for remote events in sequences.
• “remembering” information from earlier in the sequence allows them to learn long-term patterns
• dynamic, semantic information processing allows for speech recognition and language modeling

Question 18

RNNs: cons

Answer 18

1. Recursion Issues: Feedback loops can lead to numerical instability and training difficulties.
2. Scaling Challenges: Long sequences create many time steps, leading to many layers, a large number of parameters, and vanishing gradients

Question 19

How is the problem of many time points solved in RNNs

Answer 19

LSTM RNNs

Question 20

general problems with making networks bigger

Answer 20

1. larger number of parameters
2. vanishing gradients

Question 21

problem with large number of parameters

Answer 21

1. take long to train
2. many local minima
3. very sensitive to biases in training set

Question 22

what are vanishing gradients

Answer 22

• sigmoid is limited to [0,1]
• Error gets diluted across many layers, becoming smaller and smaller
• Lower layers therefore get only tiny weight changes, so they learn very slowly

Question 23

convolutional neural networks

Answer 23

• convolution helps with increasing the number of layers without exploding the number of parameters
• great for classification tasks: image/sound recognition

Question 24

training CNNs

Answer 24

standard backpropagation

Question 25

convolution

Answer 25

• core operation of CNNs
• mathematical operation where a kernel slides across an input image (or feature map) to produce an output feature map

Question 26

Why do CNNs have fewer weights to train compared to fully connected networks?

Answer 26

• Each neuron in a layer is connected only to a small local patch of the input image, rather than the entire image.
• The same kernel is reused across the entire image, reducing the number of parameters.
• i.e., only local connections, leading to fewer weights to train

Question 27

How do CNNs build feature representations?

Answer 27

• from layer to layer
• convolution is implemented with with more and more ‘complex’ kernels
• e.g., edges or colors > shapes or textures > complex features, such as object parts.

Question 28

How are CNNs similar to the brain’s visual system?

Answer 28

Both CNNs and the visual system use hierarchical, localized processing (similar to neuronal receptive fields) to build an understanding of the input.

Question 29

What activation function for CNNs

Answer 29

• ReLU (Rectified Linear Unit)
• has a steep gradient when active and no activation for negative input, making training faster.

Question 30

max-pooling

Answer 30

takes the maximum value within a small neighborhood (e.g., 2x2) in the feature map to reduce dimensions while retaining important information.

Question 31

SoftMax

Answer 31

• Converts raw scores into probabilities
• ensuring that:
  1. All outputs are positive.
  2. They sum to 1.
  3. This makes the output interpretable as a probability distribution.

Question 32

layer stacking

Answer 32

stacked convolutional layers with max pooling on top of one another, followed by fully (dense) connected layers

Question 33

describe the inner structure of a convnet

Answer 33

• after each convolutional block, a max-pooling layer reduces the size of the feature map while retaining the most important features.
• after all convolutional and pooling layers, the flattened feature maps are passed into fully connected layers.
• the final dense layer has as many neurons as there are classes and outputs outputs probabilities for each class using using softmax.

Question 34

what does training these convnets require

Answer 34

1. many parameters
2. large training sets
3. fast computers

Question 35

result of convnets

Answer 35

better than human performance on visual recognition task

Question 36

What are the benefits of using ReLU over sigmoid?

Answer 36

• avoids the vanishing gradient problem of sigmoid
• has a simple computation
• allows for sparse activation, speeding up learning.

Question 37

Why can’t backpropagation alone lead to General Artificial Intelligence (AGI)?

Answer 37

1. Backpropagation is not biologically plausible.
2. Most human learning is unsupervised, while backpropagation focuses on supervised learning.

Question 38

What does Yann LeCun say about supervised learning?

Answer 38

He describes supervised learning as “the cherry on the cake of neural networks” because most learning, including human learning, is unsupervised.

Question 39

What is Geoffrey Hinton’s argument regarding the brain’s capacity?

Answer 39

• The human brain has about 10^14 synapses but only about 10^9 seconds in a lifetime.
• There are far more parameters than data, so most learning must be unsupervised.

Question 40

Why is backpropagation considered biologically implausible?

Answer 40

1. It requires global knowledge of all gradients and weights to compute local error contributions, which is impossible in the brain.
2. Backpropagation through time (BPTT) needs error signals to “trickle down” to lower layers, which is impractical as the original input is long gone.

Question 41

What is an example of backpropagation’s limitations in achieving AGI?

Answer 41

While backpropagation enables Convolutional Neural Networks (CNNs) to perform specific tasks like image recognition, it cannot replicate the flexible learning and reasoning of the human brain, which involves unsupervised and adaptive learning.

Question 42

Geoffrey Hinton

Answer 42

‘The brain has about 10^14 synapses and we only live for about 10^9 seconds. so we have a lot more parameters than data. we must do a lot of unsupervised learning’