Week 4: Multilayer Perceptrons and Backpropagation

Question 1

Feedforward Neural Networks

Answer 1

There’s 1 input layer, hidden layer(s), and 1 output layer. Each layer is an array of neurons, with layers interconnected by links. Each link has a connection weight. The connections are one-way, going forward.

Question 2

Activation/Transfer Function

Answer 2

Activation functions take on various forms, with the main purpose of transforming the input into a desirable range of outputs.

Question 3

Symmetric Hard Limit (Signum) Transfer Function

Answer 3

f(n) = sgn(n)

Question 4

Linear Transfer Function

Answer 4

f(n) = n

Question 5

Symmetric Sigmoid Transfer Function

Answer 5

f(n) = 2/(1+e^{-2n}) - 1

Question 6

Logarithmic Sigmoid Transfer Function

Answer 6

f(n) = 1/(1+e^{-n})

Question 7

Radial Basis Transfer Function

Answer 7

f(n) = e^{-n^2}

Question 8

Backpropagation

Answer 8

This method allows weights to be updated based on the effective error for each hidden unit using gradient descent.

Question 9

Learning Curves

Answer 9

These curves measure the training, test, and validation error over the training process.

Question 10

Radial Basis Function Neural Networks

Answer 10

The input unit uses the linear transfer function. The hidden unit use the Radial Basis Function. The output unit can use any activation function.

d_j = ||x - c_j||, with x being inputs, c_j being the centres of the basis function. c_j can be preset or determined by a training algorithm. The result of the hidden node is h(d_j).

Similarities with MLP Networks:
- Both are universal approximators
- Both are nonlinear feed-forward neural networks

Differences with MLP Networks:
- RBF networks has only 1 hidden layer, but MLP networks can have more than 1 hidden layer.
- RBF networks use different basis functions from activation functions used in MLP networks.
- RBF networks compute distance between input patterns and centres while MLP networks compute inner product of input patterns and weights.
- RBF are trained with a 2-phase algorithms but MLP networks are trained with a single-phase algorithm

Question 11

Basis Functions

Answer 11

These functions are used in the hidden nodes. As d_j approaches infinity, h(d_j, \sigma) approaches 0.

Question 12

Gaussian Function

Answer 12

h(d_j) = e^{\frac{-d_j^2}{ 2 \sigma ^2}}, \sigma > 0

Question 13

Multi-quadric Function

Answer 13

h(d_j) = (d_j^2 + \sigma^2)^{0.5}, \sigma > 0

Question 14

Generalised Multi-quadric Function

Answer 14

h(d_j) = (d_j^2 + \sigma^2)^{\beta}, \sigma>0, 0 < \beta < 1

Question 15

Inverse Multi-quadric Function

Answer 15

h(d_j) = (d_j^2 + \sigma^2)^{-0.5}

Question 16

Generalised Inverse Multi-quadric Function

Answer 16

h(d_j) = (d_j^2 + \sigma^2)^{-\beta}, \sigma>0, \beta > 0

Question 17

Thin Plate Spline Function

Answer 17

h(d_j) = d_j^2 \ln(d_j)

Question 18

Cubic Function

Answer 18

h(d_j) = d_j^3

Question 19

Linear Function

Answer 19

h(d_j) = d_j