Lecture 3: Neural Networks

Question 1

Describe the concept of neural networks

Answer 1

It’s based on abstract views of the neuron. Artificial neurons are connected to form large networks. The connections determine the function of the neuron.

Question 2

How may neural networks not need to be ‘programmed’?

Answer 2

Connections can often be formed by learning rather than needing to be programmed.

Question 3

What did Alan Turing demonstrate about computers in 1936?

Answer 3

Computers have
electronic elements that
implement ‘logic gates’
and with these you can
build and run programs.
And hence compute
anything that is
computable

Question 4

Give 5 examples of logic gates

Answer 4

YES (input | output)
0 | 0
1 | 1

NOT (input | output)
0 | 1
1 | 0

AND (input a | b | output)
0 | 0 | 0
0 | 1 | 0
1 | 0 | 0
1 | 1 | 1

OR (input a | b | output)
0 | 0 | 0
0 | 1 | 1
1 | 0 | 1
1 | 1 | 1

XOR (input a | b | output)
0 | 0 | 0
0 | 1 | 1
1 | 0 | 1
1 | 1 | 0

NAND (input a | b | output)
0 | 0 | 1
0 | 1 | 1
1 | 0 | 1
1 | 1 | 0

NOR (input a | b | output)
0 | 0 | 1
0 | 1 | 0
1 | 0 | 0
1 | 1 | 0

XNOR (input a | b | output)
0 | 0 | 1
0 | 1 | 0
1 | 0 | 0
1 | 1 | 1

Question 5

What did McCulloch and Pitts decide regarding these logic gates and neuroscience? (5)

Answer 5

They claimed that the brain is a computer in the Turing sense and created their theoretical neuron;

1) The activity of the neuron is an “all-or-none” process
2) A certain number of synapses must be excited within the period of latent addition in order to excite a neuron at any time, and this number is independent of previous activity and position of the neuron
3) The activity only significant delay within the nervous system is synaptic delay
4) The activity of any inhibitory synapse absolutely prevents excitation of the neuron at that time
5) the structure of the net changes with time

Question 6

How does this idea differ from a real neuron? Why does it differ?

Answer 6

They chose to ignore it for purposes of practicality:
neurons tire and are less likely to fire after extensive activity.
An output signal is either discrete (0/1) or it is a real valued number (between 0 and 1) although this can be likened to firing rate.
ms delays in signal speed for neurons with long axons (conductivity delays are neglected).
Absolute inhibition was required for logic gates, called vito synapses, but not used as much anymore. Most inhibitory synapses weakly prevent the neuron from firing.
Also net input is calculated as the weight sum of the input signals and transformed into an output signal via a simple function (e.g threshold). In contrast there are often complex processes going on inside the neuron.

Question 7

How did it become clear that the computers built in neuroscience were quite dissimilar from the brain?

Answer 7

Because the real neurons are super unreliable; when it gets input it may fire but it also may not. It is quite a probabilistic process.

Question 8

What revolutionary aspect of the perceptron was quite similar to human thinking?

Answer 8

Error correcting learning: Carry out a task, evaluate what went wrong and correct those processes until the algorithm improves

Question 9

A more refined version of the error correcting-learning following the perceptron. What was this called and how was it more refined?

Question 10

What is meant by the generalised delta rule?

Answer 10

Error-backpropogation:

Question 11

How many layers and nodes were in the original perception? Also describe the weights

Answer 11

2- input and output layer: image input and classification output (e.g 8, dog etc). There were also binary nodes that take values between 0 or 1 and continuous weights between the input and output, initially chosen randomly.

Question 12

In a simple example, an input unit with a value of 0 and another with a value of 1 (input pattern (0 1)) feed into a single output pattern which only fires when it has a value of one. The weights are 0.4 of the 0 input unit and -0.1 on the 1 unit. Explain whether or whether not the unit fires and what its net input is.

Answer 12

Since the first input had a value of 0, 0.4 x 0 = 0. The second input had a value of 1, 1 x -0.1 = -0.1. We have a net input of -0.1, which gives an output pattern of (0), due to it being a negative value, meaning the neuron does not fire.

Question 13

The same example: An input unit with a value of 0 and another with a value of 1 (input pattern (0 1)) feed into a single output pattern which only fires when it has a value of one. The weights are 0.4 of the 0 input unit and -0.1 on the 1 unit.

How could we adjust the weights so that this situation is remedied and the spontaneous output matches our target output pattern of (1)? Formulate it in the form of a rule

Answer 13

The weight of the 0 input is irrelevant as it will have a value of 0 regardless of the weight. The weight of the 1 input must be a positive value, the exact number doesn’t matter.

Question 14

How is changing the weights so that the net input exceeds 0.0 typically carried out?

Answer 14

This can be achieved by adding a little to it such as 0.2. Typically .2 is added to all weights, ignoring those with activation 0 as these do not have any effect on the net input.

Question 15

All in all, describe the perceptron algorithm in words

Answer 15

For each node in the output layer:
– Calculate the error, which can only take the values -1, 0, and 1
– If the error is 0, the goal has been achieved. Otherwise, we adjust the weights
– Do not alter weights from inactivated input nodes
– Increase the weight if the error was 1, decrease it if the error was -1

Question 16

Describe the perceptron algorithm in rules

Answer 16

weight change = some small constant x (target activation - spontaneous output
activation) x input activation

if we speak of error instead of the “target activation minus the spontaneous output activation”, we have:
weight change = some small constant x error x input activation

Question 17

Why not make the constant larger?

Answer 17

Although you would immediately reach your goal, you also sometimes disturb other things other things that you’ve learned. You’re not just changing one pattern, you’re changing other patterns. You want to move incrementally so that all patterns get a chance to contribute to the final solution.

Question 18

Describe the perceptron algorithm as an equation

Answer 18

/\wji = u(tj - aj) ai = udjai

Question 19

Describe the perceptron algorithm in pseudocode

Answer 19

Start with random initial weights (e.g., uniform random in [-.3,.3])
Do
{
For All Patterns p
{
For All Output Nodes j
{
CalculateActivation(j)
Error_j = TargetValue_j_for_Pattern_p - Activation_j
For All Input Nodes i To Output Node j
{
DeltaWeight = LearningConstant * Error_j * Activation_i
Weight = Weight + DeltaWeight
}
}
}
}
Until “Error is sufficiently small” Or “Time-out”

Question 20

What is meant by perceptron convergence theory?

Answer 20

Theory proposed by Rosenblatt: If a pattern set can be represented by a two-
layer Perceptron the Perceptron learning rule will always be
able to find some correct weights. (This does not mean that it will always be fast or that there will always be a solution)

Question 21

How was the perceptron a big hit?

Answer 21

> Spawned the first wave in ‘connectionism’
Great interest and optimism about the future of neural networks
First neural network hardware was built in the late fifties and early sixties

Question 22

What were the limitations of the perceptron? (2)

Answer 22

Only binary input-output values

Only two layers

Question 23

How was the binary input-output values limitations of the perceptron solved?

Answer 23

Only binary input-output values was remedied in 1960 by Widrow and Hoff, the perceptron now gave values between 0-1 rather than 0/1 (allowing for gray images not just black and white images). This was called the delta rule. Originally mainly applied by engineers until it was later shown to be equivalent to the Rescorla-Wagner rule (1976) that describes animal conditioning very well.

Question 24

Why was the two layers limitation of the perceptron a problem?

Answer 24

Minsky and Papert (1969) showed that a two-layer Perceptron cannot represent certain logical functions. Some of these are very fundamental, in particular the exclusive or (XOR, Do you want coffee XOR tea?). It was shown that an extra layer was needed to represent this, but no solid training procedure existed in 1969 to accomplish this. This caused the first AI winter.

What was needed, was an algorithm to train perceptrons with more than two layers, preferably also one that used continuous activations and non-linear activation rules.

Question 25

How and when was the two layers limitation of the perceptron solved?

Answer 25

Such an algorithm was developed by
– Paul Werbos in 1974
– David Parker in 1982
– Yann LeCun in 1984
– Rumelhart, Hinton, and Williams in 1986 (When it was really noticed.)

Rumelhart, Hinton, and Williams introduced a hidden layer to the algorithm.

Question 26

If these advancements were made in the late eighties why were they not popularised until the nineties?

Answer 26

> Had to wait too long (days or weeks) for the computer to solve the problem
Didn’t have many example patterns (e.g handwritten numerals)

Question 27

What problem was posed by adding an extra layer to the perceptron and how was this solved?

Answer 27

It is straightforward to adjust the weights to the output layer, using the Perceptron rule. But how can we adjust the weights to the hidden layer if we don’t know what activation these should be? This was solved by the backprop trick: To find the error value for a given node h in a hidden layer, Simply take the weighted sum of the errors of all nodes connected from node h i.e of all nodes that have an incoming connection from node h:

dh = w1d1 + w2d2 + w3d3 + … + wndn

This is called the back-propagation of errors or error back-propagation

Question 28

What is the biological correlate of this back-propagation in the neuron?

Answer 28

There is none no known mechanism where some signal generated post-synaptically travels back into the axon and then arrives at the soma of that node or further back.

Question 29

Describe the characteristics of back-propagation (5)

Answer 29

> Any number of layers
Only feedforward, no cycles (though a more general version does allow this)
Use continuous nodes
– Must have differentiable activation rule
– Typically, logistic: S-shape between 0 and 1
Initial weights are random (landing on a good solution can be dependent on this)
Total error never increases (gradient descent in error space)

Question 30

What is meant by a differentiable activation rule and why is it necessary?

Answer 30

The activation rule used must be differentiable in the original formulation because otherwise you can’t use the derivative of that function.

Question 31

It was mentioned that continuous nodes values are S-shaped between 0 and 1. What does this mean?

Answer 31

It forms a logistic function. It approaches a linear function around net input = 0. Its rate of change (derivative) for a node with a given activation is: activation x (1 - activation). This means that for very high inputs, either positive or negative, the derivative becomes close to 0. This is important because the weight change rule has this derivative in it and that means that if your have certain extreme input values that the learning slows down enormously.

Question 32

Why keep the derivative there is it slows down learning?

Answer 32

Only having that derivative in there can slow down descent. It is attempting to find the ‘steepest’ slope (quickest way of reducing the error). Leaving the derivative out may find this slope quicker at times but it can also go up and down while the keeping the derivative means that the error can only go down or stay stable.

Question 33

Give benefits and downsides to the gradient descent

Answer 33

Error always goes down but it may take longer than other methods, it also does not guarantee high performance and does not prevent local minima (i.e there is no back-propagation convergence theorem). The learning rule is more complicated and tends to slow down learning unnecessarily when the logistic function is used.

Question 34

Describe the back-propagation algorithm in rules

Answer 34

weight change = some small constant x error x input activation

For an output node, the error is:
error = (target activation - output activation) x output activation x (1 - output activation)
i.e error x derivative of error

For a hidden node, the error is:
error = weighted sum of to-node errors x hidden activation x (1 - hidden activation)
i.e sum of hidden layers error x derivative of hidden layers error

Question 35

This back propagation took a long time however. How did the creators tinker with it to make it faster?

Answer 35

The learning rule is often augmented with ‘momentum term.’ This consists in adding a fraction of the old weight change when there is large decreases in error in order to avoid overshooting.

Question 36

What does the new learning rule look like adapted to include the momentum term?

Answer 36

weight change = some small constant x error x input activation + momentum constant x old weight change

Question 37

If j is a node in an output layer, what is the error dj?

Answer 37

dj = (tj - aj) aj(1-aj )

where aj is the activation of node j
tj is its target activation value, and
dj its error value

Question 38

If j is a node in a hidden layer, and if there are k nodes 1, 2, …, k, that receive a connection from j, what is the error dj?

Answer 38

dj = (w1j d1 + w2j d1 + … + wkjdk) aj (1-aj ) where the weights w1j , w2j , …, wkj belong to the connections from hidden node j to nodes 1, 2, …, k.

Question 39

The backpropagation learning rule (applied at time t) is:
/\wji(t) = udjai + B/\wji(t-1)

Explain what this equation means and what values are typically given to the variables

Answer 39

/\wji (t) is the change in the weight from node i to node j at time t,

The learning constant u is typically chosen rather small (e.g., 0.05).

The momentum term B is typically chosen around 0.5 (quite small). This is optional but makes it faster, remembers what we did in a previous iteration and going to add a little bit of that for a weight change.

Question 40

Describe an application of back-propagation which demonstrated its usefulness

Answer 40

NetTalk: Back-propagation’s ‘killer-app’ was a text-to-speech converter app which took english text and fed to to a speech synthesiser (difficult because english is quite irregular). This was connectionism’s answer to DECTalk, which was hardware about the size of a freezer carrying out the same function, required linguists and years of study. It was software and learned to pronounce text with an error score comparable to DECTalk in a week, just through feeding it training examples. The input was letter-in-context, output phoneme.

The input layer had 7 groups of 29 because each of these groups represent one letter in a sentence, and took in the context of the letters surrounding a particular letter. It had 80 hidden units and 26 output units (possible pronunciations of the letter).

Question 41

What disadvantages did back-propagation have?

Answer 41

Learning is slow, new learning will rapidly overwrite old representations, unless these are interleaved (i.e., repeated) with the new patterns. This makes it hard to keep networks up-to-date with new information (e.g., dollar rate). This also makes it very implausible from as a psychological model of human memory

Question 42

What advantages did back-propagation have?

Answer 42

> Easy to use
– Few parameters to set
– Algorithm is easy to implement
> Can be applied to a wide range of data
> Is very popular
> Has contributed greatly to the ‘new connectionism’ (second wave)