ANN

Question 1

What is the difference between supervised ANNs and unsupervised ones?

Answer 1

Supervised = trained with labelled inputs - weights attempt to fit correct output - inductive

Unsupervised = learning done iteratively to satisfy some learning rule

Question 2

What are the types of ANN topologies?

Answer 2

Feed forward: all weights are directed forwards

Recurrent: Weights can point backwards and provide immediate feedback.

[Hidden] Single / Multiple layered

Partially / Fully connected: describes connectivity between nodes across layers

Question 3

What are the different activation functions? Explain them.

Answer 3

Linear: output = c*input
Threshold: If weighted sum > 0, output = 1; else -1;
Sigmoid: Continuous threshold. Bound output between 1 and -1;

Question 4

When would you use ANNs?

Answer 4

When:

1. Input data is high dimension + continuous
2. Data is noisy
3. Long training times are OK
4. When you have enough labelled training data
5. When target function is unknown
6. When explaining the result is not important

Question 5

What are the 3 learning rules? Describe them.

Answer 5

1. Hebb’s rule: If 2 connected neurons are simultaneously on => weight( new ) = weight (old) + x1x2

2. Perceptron rule: weight (new) = weight(old) + σ(t-o)x
t = target output (0,1)
o = actual output (0,1)
x = neuron input
σ = learning rate
*** threshold activation

3. Delta rule: weight (new) = weight(old) + σ(t-o)x
   * ** linear/continuous activation
   * ** outputs can be anything

Question 6

What are perceptrons?

Answer 6

Linear classifiers - a hyper plane ( vector space of one dimension lower that divides a space classifying data points )

If weighted sum + bias > the boundary, perceptron outputs a 1

Else, 0.

*** outputs can only be 1 or 0/-1

Question 7

What is the point of the bias term?

Answer 7

Speeds up learning by shifting the hyper plane from the origin - do not always need them

Question 8

What are some of the properties of perceptrons?

Answer 8

Can classify inputs as 0 or 1 => can simulate any logic gate

Question 9

How does perceptron learning work?

Answer 9

Using the perceptron learning rule:
For N training examples:
if o /= t:
Update weights according to the learning rule s.t error is minimised
Stop when error is acceptable or i = N
Note: perceptron rules adapts weights only

Question 10

What is the fundamental basis of perceptron learning?

Answer 10

Error correction learning => adjust weights until o = t

Question 11

What is the limitation of single perceptrons? What is the solution?

Answer 11

Can only classify linear separable spaces

Multiple Layer perceptrons = interconnected perceptrons

Question 12

When use the delta learning rule?

Answer 12

If the range of values we want to be able to produce is continuous - continuous activation functions

*** t and o do not have to be 1 or 0

Question 13

How does delta learning work?

Answer 13

The exact same way as perceptron learning

Question 14

DLR vs PLR - differences and similarities

Answer 14

Same: both learn through error correction

Different:

PLR can only work with threshold activation functions (0;1)

DLR can work with any differentiable activation function

Question 15

What is a universal function approximator?

Answer 15

ANN with at least one hidden layer (1) enough nodes (2) continuous activation function (3) can approximate any continuous non-linear function

Question 16

What is the purpose of the delta learning rule?

Answer 16

Classify non linearly separable problems

Question 17

How do you decide the type of perceptron organisation to use as well as the activation function?

Answer 17

Is the problem linearly separable?
No: use MLP
Yes: Is the output Binary?
 No: use single perceptron + continuous 
 activation function
 Yes: use single perceptron + threshold activation 
 function.

Question 18

What is the weight/hypothesis space?

Answer 18

Mapping of weights to error - gradient descent aims to find minimums in this topology

Question 19

How do we calculate error for learning by error minimisation for the ∆ rule?

Answer 19

E(w) = 1/2Σ(t-o)^2 ==> modified MSE

Question 20

What is gradient descent?

Answer 20

Iteratively change weights to reduce MSE

Can implement DLR at a neuron or network level.

Always moves us in the direction of the steepest decrease in slope

Question 21

What is backpropagation?

Answer 21

Process of feeding error back through the network to determine by how much we need to adjust weights.

Question 22

What do we need for back-propagation to work?

Answer 22

1. differentiable activation function

2. non linear activation function

Question 23

What is the back propagation algorithm?

Answer 23

1. Forward pass
2. Calculate ẟi for output neurons
   [ẟi = Oi*(1-Oi) * (Ti-Oi)] - sigmoid
3. Calculate change in weights to output nodes
   [∆Whi = η * ẟi * xhi] - where xhi is the output from the previous node
4. Calculate ẟh for hidden neurons
   [ẟh = Oh * (1-Oh) * ΣWhi*ẟi]
5. Calculate change in weights to hidden nodes
   [∆Whi = η * ẟi * xhi] - where xhi is the output from
6. Update weights