5 NDC Feedforward Rate Networks and Backpropagation

Question 1

How does an INDIVIDUAL rate neuron model perform simple computations?
What does this computation depend on?

Answer 1

by transferring its input into its desired output
depends on weight and threshold

Question 2

What is the purpose of a linear decision boundary in rate neuron models?

Answer 2

to categories inputs into TWO group - one group elicits strong response

Question 3

What is the shape of the response curve in rate neuron model?

Answer 3

sigmoidal

Question 3

What type of neuron is in the Leaky Integrate and Fire model?
Are the dynamics of this model linear, quadatic or cubic?

Answer 3

-a single point neuron
-linear dynamics

Question 4

What is the rate in the rate neuron model represent?

Answer 4

graded activity / firing rate ??

Question 5

What does tau represent in the sigmoidal response curve of the rate neuron model?
As tau changes, what happens to this curve?

Answer 5

-parameter that characterises time scale
-smaller tau -> steeper sigmoidal curve and vice versa

Question 6

What does u represent in the sigmoidal response curve of the rate neuron model?

Answer 6

-strength of input neuron receives

Question 7

What type of values go in a truth table?

Answer 7

binary: 1 and 0

Question 8

In a linear rate neuron model with two inputs, what is the equation?

And also with a threshold?

Answer 8

y=x1w1 + x2w2

theta < x1w1 + x2w2

Question 9

In a rate neuron model as you change operation from AND to OR, what happens to the threshold (theta)?

Answer 9

threshold hold decreases

Question 10

What kind of equation is the decision boundary of rate neuron model?
What is the equation? what is m?

Answer 10

-linear (equation of a straight line)
-x2=mx1+c
m is gradient

Question 11

When theta equals zero, where does the (linear) decision boundary go through?
Why?

Answer 11

theta =0, goes thru origin of axes

because theta represents the the y intercept c

Question 12

What are the three different activation functions? and their curve shapes?

Answer 12

linear - I (linear)
threshold - squarery S
logistic sigmoid - S

Question 13

When is the response of the rate neuron model greatest?

Answer 13

when the pattern of inputs matches the pattern of weights

Question 14

In the logisitic activation function eqn., what is w3 equal to?

Answer 14

minus theta

Question 15

What is supervised learning?

Answer 15

learning from example or from a teacher

Question 16

What is the point of using stochastic gradient descent in the rate neuron model?

Answer 16

to iteratively adjust the weights in the direction which reduces the error

Question 17

What is the equation for gradient descent?
What does it mean?

Answer 17

𝐞𝐫𝐫𝐨𝐫[𝐨𝐮𝐭𝐩𝐮𝐭]=(target−𝐨𝐮𝐭𝐩𝐮𝐭)^2

to get the error of the output, find the difference between the desired/target output and the output we have now (and the square it)

Question 18

What is the output of a rate neuron model ? y=

Answer 18

weighted sum of inputs
∑wi xi

Question 19

What is the learning rate in the gradient descent recipe? What does it do?
Why can’t it be too big or too small?

Answer 19

alpha: is the magnitude of the adjustment of the weights/ the size of the steps

big: overfit of weights an miss optimal low error points
small: can cause the model to take forever to adjust to the ideal weight

Question 20

Chain Rule:
If we have a function A[B] of a function B[C],
finding the derivative of A[B[C]] with respect to C is given by what?

Answer 20

𝑑𝐴[𝐵[𝐶]]/𝑑𝐶 = 𝑑𝐴/𝑑𝐵×𝑑𝐵/𝑑𝐶

Question 21

In the gradient descent graph, what are the axes?

Answer 21

y = error[wi] x=wi

Question 22

As the learning rate (alpha) decreases what happens to the speed of the descent ?

Answer 22

gradient descent is faster (smaller steps)

Question 23

What is the chain rule of the gradient descent?

Answer 23

derror[output[weight]] / dwi

Question 24

What is the gradient descent recipe?

Answer 24

• Identify the variables you need to change; 𝒘_𝒊 for a neuron.
• Define a cost function that you need to minimise to get the desired output:
  The squared error, the squared difference between the neuron’s actual output and desired output.
• Use the chain rule to find the derivative of the cost function with respect to the variable being updated; Δ𝑤_𝑖
  The derivative tells you how much change in the error will result from a change in 𝑤_𝑖
  Moving in the direction of the derivative will increase the error; move in the opposite direction.
  Multiply the derivative by a learning rate to adjust the step size of the learning rule.
  𝑤_𝑖←𝑤_𝑖−𝛼⋅Δ𝑤_𝑖

Question 25

what is the truth table like for the XOR exclusive or computation?

Answer 25

neuron response (output) given when light or bell is on but not both at same time

Question 26

What are hidden neurons?

Answer 26

neurons behind output neuron in feedforward networks

Question 27

What sort of network do you use to solve the XOR problem?
What computation are the left and right hidden neurons doing?

Answer 27

-feedforward neural network
-left: AND
right: OR