Neural nets 2

Question 1

What problems can SLPs not solve?

Answer 1

XOR problems (non linearly separable problems)

Question 2

What is the difference between SLPs and multi-layered perceptrons?

Answer 2

MLPs have hidden layers between the input and output nodes.

Question 3

What is the benefit of a hidden layer?

Answer 3

A hidden layer of neurons
can transform a non-linear
problem into a number of
simpler linearly separable
problems

Question 4

Name a problem with training MLPs

Answer 4

Credit assignment problem. Which weights cause the problem? Hidden->output layer weights or input->hidden layer weights?

Question 5

How do you train MLPs?

Answer 5

target (t) – output (y) error/cost term

Question 6

What is backpropagation used for?

Answer 6

Backpropagation is used to solve the credit assignment problem. We want to back-chain
the error from the output layer to the input layer to update the weights of the network layer by layer

Question 7

What are the 8 backpropagation algorithm steps?

Answer 7

1. Present the data vector at the
   input layer,
2. Pass weighted input layer
   activations to hidden layer
3. Pass weighted hidden layer
   activations to output layer
4. Apply the target value to the
   output layer
5. Calculate the δ 𝑜𝑜 output error
   values
6. Update the hidden-output layer
   weights using using the δ 𝑜𝑜 values
7. For each hidden node, calculate its
   δℎ error value using δ 𝑜𝑜.
8. Update the input-hidden layer
   weights using the δℎ values.
   Step 1-3 Forward Pass
   Step 4-6Backward Pass

Question 8

But how do we find the delta (error) values for each layer?

Answer 8

*Use gradient descent
principle
*Use the chain rule (from
calculus)

Question 9

What is a problem with the gradient descent principle?

Answer 9

We can get stuck in a local minima. We would like to find the global minimum but with gradient descent we can’t be sure we have.

Question 10

How do we solve the credit assignment problem?

Answer 10

1) Backpropagating (or backchaining) network error (and proxy error)
terms through the layers of weights in the network using calculus;
2) All nodes’ activation from the forward pass is maintained in order to appropriately assign error to the corresponding weights (if the
weights aren’t connected to active nodes they can’t have caused the error)