Chapter 7: Artificial Neural Networks

Question 1

Give the three types of neural network models

Answer 1

single neuron model

single layer perceptron

multilayer perceptron

Question 2

how is the output of a neuron created

Answer 2

activation function * ( sum of (weights * input) + bias )

Question 3

what does an activation function do

Answer 3

squishes the amplitude range of the output signal

Question 4

give some activation functions

Answer 4

identity

threshold

sigmoid/ tanh

Relu

Question 5

what is the identity function

Answer 5

no change φ(v) = v

Question 6

what is the symbol for the activation function

Answer 6

phi φ

Question 7

what is the threshold function

Answer 7

maps to {-1 , 1}
φ(v) = 1 if v >= 0
-1 if v < 0

Question 8

what is the sigmoid function

Answer 8

maps to { 0 , 1 }

1 / 1 + exp (-v)

Question 9

give the tanh sigmoid function

Answer 9

maps to { -1, 1}
exp( 2v ) - 1
—————-
exp( 2v ) + 1

Question 10

give the rectified linear unit function

Answer 10

maps to ( v, 0 )
φ(v) = v if v >= 0
0 if v < 0

Question 11

what does relu stand for

Answer 11

rectified linear unit

Question 12

describe a single layer perceptron

Answer 12

one input layer

one output layer

Question 13

what are hidden layers

Answer 13

between the input and output

creates complex function

Question 14

what is a multi layer perceptron

Answer 14

also called a feed forward neural network

consists of at least three layers

Question 15

what are the hyperparameters in a neural network

Answer 15

the number of hidden layers

the number of neurons in each layer

the number of hidden layers

Question 16

what is training in neural networks

Answer 16

the process of finding the optimal settings for the weights

Question 17

what are the two training methods for a neural network

Answer 17

Hebbian learning

gradient descent

Question 18

what is Hebbian learning

Answer 18

calculate weights based on the values of the nodes at either end
wij = learning rate * xi * xj

Question 19

how is gradient descent different in a neural network

Answer 19

we cant set to 0 as this is an unsolvable equation

Question 20

what is the perceptron algorithm

Answer 20

train a single neuron by minimising the perceptron criterion using SGD.

The activation function is identity and the weights and bias equivalent to coefficient vector of linear model

Question 21

what is the perceptron criterion (describe)

Answer 21

how we minimise the error function in the perceptron algorithm

Question 22

give the equation of the perceptron criterion

Answer 22

O(w) = - sum yi (W^T Xtilde i)

Question 23

how do we minimise the perceptron criterion

Answer 23

SGD weight update

Question 24

give the equations for SGD weight update

Answer 24

Oi(w) = -yi W^T Xtildei
derivative respect to w = -yixi

hence
w^(t+1) = W^t + learningrate * yi * xi

Question 25

what is a feed forward netural network

Answer 25

calculate error front to end

Question 26

what is cross entropy loss

Answer 26

plug a logistic regression model into the prediction (output) layer
(e.g. softmax)

Question 27

how do we regularise a neural network

Answer 27

add a regularisation term to the final optimisation objective function for training

Question 28

give the equation for l2 regularisation

Answer 28

O(wnn) = loss(wnn) + lamda * 0.5 * || wnn || 22

Question 29

describe the steps of backpropagation

Answer 29

1. calculate loss function (feed forward)
2. expand to get a full equation for each output node
3. calculate the sum of least squares error
4. apply the chain rule