Lecture 14 - Neural Networks Part 2

Question 1

How is learning achieved in a MP Neuron?

Answer 1

For each example in a training set:

After firing, adjust the weight of each input to try to get the desired output.

Question 2

How can the threshold of a MP Neuron be modified on-the-fly?

Answer 2

Set the threshold to 0 and add an adaptive input that acts as the threshold instead

The value of this threshold-input can be changed at the same time as the weights

Question 3

What is the general formula for learning in MP Neurons?

Answer 3

New weight = old weight + change in weight

Question 4

What is the Hebb rule?

Answer 4

Learning can take the form of strengthening the connections between any pair of neurons that are simultaneously active

Question 5

What is the formula for the Hebb rule, as applied to MP Neurons?

Answer 5

w_i(t+1) = w_i(t) + ax_iz

where a is a constant that determines the rate of learning

z is the desired output

x is the current input

Question 6

What is the perceptron rule?

Answer 6

Adjust weights like the Hebb rule, but only if the current weights would give the wrong answer

i.e. Hebb rule with error correction

Question 7

What is the advantage of the perceptron rule?

Answer 7

Provided the function determining z is linearly separable and a small a is chosen, the rule will converge on a set of weights that give the correct rule

Question 8

What is the drawback of the preceptron rule?

Answer 8

No convergence will occur if the data is not linearly separable

Question 9

What is the Delta Rule also known as?

Answer 9

Widrow-Hoff rule

Question 10

What is the benefit of using the Delta rule over the perceptron rule?

Answer 10

Converges even if the set is not linearly separable

Question 11

What is the formula of the delta rule?

Answer 11

w_i(t+1) = w_i(t) + ax_i(z-y)

Question 12

Suppose z = f(x1….xn)

The weights of the MP neuron will adapt to values such that the unit provides a linear approximation to _______

Answer 12

the function f

Question 13

For function approximation, it can be shown that the delta rule is an ______ strategy for changing the weights

Answer 13

Optimal

Question 14

What is a feed forward neural network?

Answer 14

Signals flow in one direction through layers of neurons

Question 15

When is a neural network “deep”?

Answer 15

When there is more than one hidden layer

Question 16

Why is there no point building a multilayer feedforward network if the units perform linear transformations of their inputs?

Answer 16

The weight matrix of a multilayer system is the same as the weight matrix of a single layer system if the neurons perform linear transformations

Question 17

What are the two major limitations of the delta rule?

Answer 17

It can only learn linear functions

It is equivalent to linear regression but much much slower than normal algorithms and added complexity with choosing a