05 Neural Networks

Question 1

The Learning Problem

Answer 1

Improve over task T with respect to performance emasure P based on experience E.

Question 2

Supervised learning

Question 3

Unsupervised learning

Answer 3

Training data does not include desired outputs, instead the algorithm tries to identify similarities between the inputs that have something in common are categorised together.

Question 4

Reinforcement learning

Answer 4

The algorithm is told when the answer is wrong, but does not get told how to correct it. Algorithm must balance exploration of the unknown environment with exploitation of immediate rewards to maximize long- term rewards.

Question 5

Evolutionary learning

Answer 5

Biological organisms adapt to improve their survival rates and chance of having offspring in their environment, using the idea of

fitness (how good the current solution is

Question 6

The Machine Learning Process

Answer 6

1. Data Collection and Preparation
2. Feature Selection and Extraction
3. Algorithm Choice
4. Parameters and Model Selection
5. Training
6. Evaluation

Question 7

We are born with about _____ neurons. A neuron may connect to as many as _____ other neurons

Answer 7

We are born with about 100 billion neurons. A neuron may connect to as many as 10,000 other neurons

Question 8

Hebb’s Rule

Answer 8

• Strength of a synaptic connection is proportional to the correlation of two connected neurons.
• If two neurons consistently fire simultaneously, synaptic connection is increased (if firing at different time, strength is reduced).
• “Cells that fire together, wire together.”

Question 9

How realistic is McCulloch and Pitts Neurons Model?

Answer 9

Not Very.

– Real neurons are much more complicated.

– Inputs to a real neuron are not necessary summed linearly.

– Real neuron do not output a single output response, but a SPIKE TRAIN.

– Weights wi can be positive or negative, whereas in biology connections are either excitatory OR inhibitory.

Question 10

Neural Networks: Updating the weights

Answer 10

Aim: minimize the error at the output

Question 11

The learning rate n

Answer 11

ɳ controls the size of the weight changes.

• Why not ɳ = 1?
– Weight change a lot, whenever the answer is wrong. – Makes the network unstable.

• Small ɳ
– Weights need to see the inputs more often before they change significantly.
– Network takes longer to learn.
– But, more stable network.

Question 12

Bias Input

Answer 12

• What happens when all the inputs to a neuron are zero?
– It doesn’t matter what the weights are,
– The only way that we can control whether neuron fires or not is through the threshold.

• That’s why threshold should be adjustable.
– Changing the threshold requires an extra parameter that we need to write code for.

• We add to each neuron an extra input with a fixed value.

Question 13

A single layer perceptron can only learn _____ problems.

Answer 13

A single layer perceptron can only learn linearly separable problems.

Boolean AND function is linearly separable, whereas Boolean XOR function (and the parity problem in general) is not.

Question 14

In contrast to perceptrons, multilayer networks can learn not only multiple _______, but the boundaries may be _____.

Answer 14

In contrast to perceptrons, multilayer networks can learn not only multiple decision boundaries, but the boundaries may be nonlinear.

Question 15

Linear Models can only identify flat decision boundaries like ___

Answer 15

Linear Models can only identify flat decision boundaries like straight lines, planes, hyperplanes, …

Question 16

MLP

Answer 16

Multi-Layer Perceptron

Question 17

The multilayer network structure, or architecture, or topology, consists of ____

Answer 17

The multilayer network structure, or architecture, or topology, consists of an input layer, one or more hidden layers, and one output layer.

Question 18

A network with ____ layers of _____ is a three- layer network

Answer 18

A network with two layers of hidden units is a three- layer network

Question 19

Properties of the Multi-Layer Network

Answer 19

• Layer n-1 is fully connected to layer n. • No connections within a single layer.
• No direct connections between input and output layers.
• Fully connected; all nodes in one layer connect to all nodes in the next layer.
• Number of output units need not equal number of input units.
• Number of hidden units per layer can be more or less than input or output units.

Question 20

What Do Each of The Layers Do?

Answer 20

Question 21

How to learn Multi Layer Perceptrons?

Answer 21

Backpropagation

Question 22

Backpropagation

Answer 22

1. Calculate the output errors
2. Update last layer of weights
3. Propagate error backward, update hidden weights
4. Until first layer is reached

Question 23

The backpropagation training algorithm uses the ____ technique to minimize the_____ between the desired and actual outputs.

Answer 23

The backpropagation training algorithm uses the gradient descent technique to minimize the mean square difference between the desired and actual outputs.

Question 24

MLP is trained initially selecting____ weights and then presenting all training data incrementally.

Answer 24

MLP is trained initially selecting small random weights and then presenting all training data incrementally.

Question 25

Gradient Descent in MLP (figure and equation)

Answer 25

Question 26

Update rules for MLP

Answer 26

Question 27

MLP: What do we want in an activation function?

Answer 27

• Differentiable
• Nonlinear (more powerful)
• Bounded range (for numerical stability)

Question 28

Sigmoidal function

Answer 28

Sigmoidal (logistic) function

g(a_i) = 1 / (1 + exp(-ka_i)

Question 29

MLP: Learning capacity for different number of layers

(with sigmoid activation function)

Answer 29

Question 30

MLP: Selecting initial weight values

Answer 30

• The MLP algorithm suggest that weights are initialized to small random numbers (<± 1), both positive and negative
• Choice of initial weight values is important as this decides starting position in weight space. That is, how far away from global minimum
• Aim is to select weight values which produce midrange function signals (not in only saturated signal, see sigmoid function)
• Select weight values randomly from uniform probability distribution
• Normalise weight values so number of weighted connections per unit produces midrange function signal

Question 31

MLP: When should the weights be updated?

Answer 31

After all inputs seen (batch)
• More accurate estimate of gradient
• Converges to local minimum faster (Jim doesn ́t agree!) •

After each input is seen (sequential)
• Simpler to program and most commonly used
• May escape from local minima (change order or presentation)
• Both ways, need many epochs - passes through the whole dataset

Question 32

MLP: Input Normalization

Answer 32

• Stops the weights from getting unnecessarily large.
• Treat each data dimension independently.
• Each input variable should be processed so that the mean value is close to zero or at least very small when compared to the standard deviation.

Question 33

MLP: Rule of thumb for amount of training data needed

Answer 33

10 times more data than the number of weights

Question 34

Overfitting

Answer 34

Overfitting occurs when a model begins to learn the bias of the training data rather than learning to generalize.

Overfitting generally occurs when a model is excessively complex in relation to the amount of data available.

A model which overfits the training data will generally have poor predictive performance, as it can exaggerate minor fluctuations in the data.

When we fit the model, it cannot tell which regularities are relevant and which are caused by sampling error.
– So it fits both kinds of regularity.
– If the model is very flexible it can model the sampling error really well. This is not what we want.

Question 35

Solution to overfitting

Answer 35

(k-fold) Cross-validation

Question 36

Validation data

Answer 36

Question 38

k-fold cross validation

Answer 38

• Divide all data into k sets
• For i = 1…k:
  
  • Train on data[i], validate on data[i + 1], test on rest
• Average the results