Computer Practicals

Question 1

In the lecture, you saw the following equation:

ai=f(∑ wij aj−bi)

Which aspects of the biological neuron are being modelled by the variables?

Answer 1

wij : Weight, so the strength of the synapses j to neuron i

bi: Bias, a value which determines how quickly a neuron will respond to particular inputs.

f: Transfer function, so the transduction of the incoming input to the output of the neuron.

ai: Output of the neuron i, in firing rate.

Question 2

Which physiological aspects of real (biological) neurons are missing in this model?

Answer 2

Several aspects of real neurons are missing in this model, because this model is a gross oversimplification. Some examples:

Timing of spikes, there is no time component in this model.
Abundancy of dendrites / axon branches and their spatial arrangement.
Refractory period.
Adaptation, plasticity.
No biomolecular effects such as aging, hormones, disease etc.
A biologically plausible transfer function; a hardlimit transfer function is not possible in real life.
Effects of LTP.

Question 3

Consider a neuron, which receives input from a single presynaptic neuron. We will now drop the subscripts and call the output of the neuron of interest a, the output of the presynaptic neuron p, and the bias b . consider two settings, namely w=1 and w=−1, and b=0.5 with a hard-limit function (hardlim) to start with.

Describe how output a varies as a function of p (where p is between 0 and 1), for both cases w=1 and w=−1.

Answer 3

w=1: This neuron simply multiplied the input p by 1, thus doesn’t change it. If this reached threshold b=0.5 it fires, i.e. the output a=1

w=−1: This neuron received negative input from the negative w, so this can be seen as inhibitory input. With p between 0 and 1, it will never reach threshold and spike (i.e. never reach a=1)

Question 4

Assume now that the input gets a larger dynamic range and can vary between -4 and 4.

Describe again how the output a varies as a function of p .

Answer 4

When pp is allowed to vary from -4 to 4:
w=1: As before, the in input pp is multiplied by 1, thus stays the same. Ergo, when p reached threshold b=0.5 the cell will spike, as in a=1.

w=−1 : The input is multiplied by w=−1. In this case, a negative input will be able to reach threshold, i.e. p< −0.5. Other inputs will not reach threshold.

Question 5

We further assume that we receive a binary signal as input (p is either 0 or 1), however now we can vary the bias, b (-2,2).We again study both cases w=1 and w=−1 . For what ranges of the bias b will our neuron be able to discriminate a binary input (which takes on either 1 or 0) coming from the pre-synpatic cell?

Answer 5

w=1: Here the best classification between 1 and 0 is the middle, i.e. b=0.5. As the weight won’t affect the p value, inputs of 0 will be < b=0.5 and inputs of 1 > b, giving appropriate outputs. If the input will only be 0 or 1, b should be in the range 0≤b<1
w=−1w=−1: When the input is multiplied by −1−1, it’s basically the same but negative, ergo bb should be in the range −1≤b<0.

Question 6

What is the output of the hardlim function?

Answer 6

0 or 1

Question 7

Give two alternative functions given in the first assignment

Answer 7

Linear function: purelin(x): f(x) = x
Hyperbolic tangent: tanh(x): f(x) = e^2x - 1 / e^2x + 1

Question 8

What are the codomains of these two functions?

Answer 8

The codomain of the linear function is (−∞,∞), while for the hyperbolic tangent it is (−1,1).

Question 9

Describe the graphs given by these two functions

Answer 9

Linear function: Gives a linear graph with a continuous output
Hyperbolic tangent: Gives an S shaped curve between -1 and 1 on y axis

Question 10

What effect does the bias have on these two graphs

Answer 10

Linear function: Shifts the line along the x axis
Hyperbolic tangent: Shifts the line along the x axis

Does not change the shape of the function, just the amount required for the same output, higher bias means more shifted to the right and more input required for a higher output

Question 11

Which configuration of transfer function and parameters classifies values of the input pp most strictly into ‘categories’? We refer to transfer functions that make such a strong distinction between two ranges of input values as classifiers.

Answer 11

To make a strong classifier, the hardlimit transfer function would be ideal. The values of b and w will depend on what you want to classify, but the w can’t be 0. Dependent on your input p, your threshold should be of a logical value, i.e. inputs and weight of 1 and a threshold of 500 will not classify.
For instance, as seen in previous questions, classifying binary inputs to binary outputs (a really boring classification), can be readily done with 0≤b<1, and a w=1

Question 12

Conversely, which configuration is the worst at grouping values of the input in separate categories?

Answer 12

A linear transfer function (like purelin) will not give strict classifications, but a continuous output.

Question 13

What could be the advantage of a poorly classifying configuration?

Answer 13

This will be an estimator. If you want your output to reflect the input in a continuous, linear fashion, you would need this configuration.

Question 14

Which transfer function is the most biological one? Explain!

Answer 14

Of our transfer functions the hyperbolic tangend (tanh) function is the most biologically plausible one. These shapes of curves are often seen in biology and psychology

Question 15

In which input range does the hyperbolic tangend (tanh) transfer function behave as an estimator of the input, and in which range as a classifier?

Answer 15

Estimator range: The middle of the tansig curve is relatively straight, and acts as an estimator, much like a linear transfer function.
Classifier range: The ends of the tansig curve plateau towards 0 and 1, and this acts similar to the hard limit transfer function.

Question 16

What is the essential mathematical feature of a transfer function aimed at classification?

Answer 16

The feature of non-linearity; it needs a relatively strict distinction between values, such as we see in the hard limit function. This is highly non-linear.

Question 17

Which aspects of how neurons render the transfer from synaptic input to output can be modelled with a sigmoid function?

Answer 17

1. The sigmoid function has a threshold-like point, it has to reach certain input to start responding. 2. Moreover, in the middle, a small increase in input gives a large increase in output (i.e. it has a dynamic range). 3. furthermore, the function also has a maximum, where the output doesn’t increase further.

Question 18

Which physiological aspects of real neurons can be seen as the counterpart of the bias in computational neurons?

Answer 18

This can be thought of as the resting potential of a neurons, spontaneous input of other neurons on said neuron, biomolecular effects (i.e. hormones etc), I also said LTP

Question 19

What is meant bu Boolean algebra?

Answer 19

Data with only two possible values (think of a neuron that can only produce output which is either 1 or 0) is handled by a branch of mathematics called Boolean algebra, where the values 1 and 0 are often replaced by the labels true and false.

Question 20

Boolean algebra comes with a set of basic operations or functions, which take as input boolean values and produce boolean output. Give three examples of these

Answer 20

AND, OR, XOR:
OR function, which given two boolean inputs, returns True if at least one of the inputs is True, or the exclusive or (XOR) function, which returns True only if exactly one of the two inputs is true. AND returns true only if both are true

Question 21

For what inputs of x and y would activate these three functions?

Answer 21

Can be visualised with a truth table:

X | Y | X and Y | X or Y | X xor Y
0 | 0 | 0 | 0 | 0
1 | 0 | 0 | 1 | 1
0 | 1 | 0 | 1 | 1
1 | 1 | 1 | 1 | 0

Question 22

How are these 1s and 0s represented in Python and other coding languages?

Answer 22

True

Most programming languages implement a boolean data type, which can be represented by 0 and 1 or by True and False labels. In Python, True and False are built-in constants, and we can ask Python to combine them with Boolean operations:

a = True
b = False
print(a or b)

We can also use the numeric values 1 and 0, as long as we tell Python that we mean boolean and not integer values.

a = bool(1)
b = bool(0)
print(a and b)

Question 23

In electronics, what is a logic gate?

Answer 23

In electronics, a logic gate is a (conceptual) device which implements a boolean function. We can think of neurons as biological logic gates, in the sense that we can consider our modelled neuron as a system which receives binary inputs (e.g. spike or no spike from its presynpatic neurons) and outputs a binary signal (spike or no spike).

Question 24

How do we broadly programme a neuron at the beginning of assignment 2?

Answer 24

We begin by writing a very simply Python class which we call neuron. This neuron has two methods, namely __init__, which is used to generate instances of this class, and output, which produces the neuron’s output given the inputs.

Question 25

Write down the general equation that describes output a as function of the inputs pi for some transfer function f. Then, simplify the equation for the case of a linear transfer function and two inputs p1 and p2 .

Answer 25

General equation:
a = f(∑ wi pi - b)

Simplification:
a = f(∑ wi pi - b)
a = f((w1 p1 + w2 p 2) - b) only two inputs
a = w1 p1 + w2 p 2 - b linear transfer function

Question 26

Given a neuron with 2 inputs and a hardlim function, computing the output using the expression for the linear transfer function, and whenever it is larger or equal to zero we set it to True, and otherwise to False. The parameters are set to bias=1, weight_1=1, weight_2=-1. Fill out a truth table corresponding to the boolean operation performed by this neuron.

Answer 26

1 | 2 | neuron
0 | 0 | 0
1 | 0 | 0
0 | 1 | 0
1 | 1 | 0

Question 27

Describe why there is no output for this neuron

Answer 27

Since the second weight is set to a negative value, and the input can only take on a positive value, the output of this neuron can only be a negative value. In the hardlim function, the bias can act as the threshold. Since it is set to 1 and the output < 0 the threshold cannot be reached.

Question 28

What values of the bias and the weights can the output of our neuron become equivalent to the AND function shown previously? (bias can be between 0 and 1; weights can be between -2 and 2 in steps of 0.5)

Answer 28

Bias = 1
W1 = 1
W2 = 0.5

Bias = 1
W1 = 0.5
W2 = 1

Bias = 1
W1 = 1
W2 = 1

If the weights are any less (e.g both 0.5) then their sum does not exceed 1. If the bias = 0.5 then the weights must be 0.5

Question 29

Now find parameter values such that the computation performed by the neuron corresponds to the OR function.

Answer 29

Bias = 0, 0.5
W1 = 1
W2 = 1

As long as one input fires, it should surpass the threshold

Question 30

Now try find parameter values such that the computation performed by the neuron corresponds to the XOR function.

Answer 30

You can’t

Question 31

Can you show mathematically why this is not possible?

Answer 31

The neuron output is defined as: w1∗p1+w2∗p2=a
For the neuron to apply XOR, it should follow these constraints:
1 | 2 | neuron
0 | 0 | 0
1 | 0 | 1
0 | 1 | 1
1 | 1 | 0

Following conditions (2) and (3) we get:

w1=1w1=1
w2=1w2=1
Which, when substituted in (4) gives:

1∗1+1∗1=0
Thus the system cannot be solved.

In general, substituting any 2 values in all 4 equations shows that this set of equations cannot be true at the same time.

Question 32

What is meant by the decision boundary?

Answer 32

Another way to think about what the neuron is doing is to plot each combination of input values on a 2-D plane. Then, the parameters of our neuron determine a line on this 2-D plane that will separate points in input space for which the neuron becomes active (responds with a 1) from points where the neuron remains inactive (responds with a 0). This line is known as the decision boundary.

Question 33

Derive the equation for the line which represents the decision boundary, through expressing p_2 as function of p_1. (this will allow you to visualise the boundary)

Answer 33

Starting with:

a=w1p1+w2p2−b
p1 = w2p2 + w1−b-a
Since the transfer function is a hardlim, it will be activated from zero and as such:

0=w1p1+w2p2−b
b=w1p1+w2p2
b−w1p1=w2p2
(b−w1p1)/w2=p2

Question 34

What would the decision boundary have to look like for the XOR operation to be carried out? Give a graphical explanation of why XOR cannot be solved by a single linear classifier.

Answer 34

The decision boundary would have to be laid out in a manner that activated the bottom left unit as well as the top right unit, however this is not possible with only one linear classifier as there is no line that could be drawn which could only enclose these. A curved decision boundary is necessary. Not linearly separable.

Question 35

How was this problem solved?

Answer 35

More neurons: the strength is in the union of many computationally simple elements. You can solve this decision boundary plot with two lines

Question 36

Translate this decision boundary plot into truth table entries by filling in the Output neuron 1 and Output neuron 2

Answer 36

There are two solutions. The first is to have one neuron compute an OR-function and the other a NOT-AND operation. The final computation by the second layer is an AND operation:

1 | 2 | Output1| Output2 | Output3
0 | 0 | 0 | 1 | 0
1 | 0 | 1 | 1 | 1
0 | 1 | 1 | 1 | 1
1 | 1 | 1 | 0 | 0

The second solution is to have one neuron is only activated by Input 1 (negative weight input 2) and the other only by Input 2 (negative weight input 1). The third neuron can perform an OR operation:

1 | 2 | Output1| Output2 | Output3
0 | 0 | 0 | 0 | 0
1 | 0 | 0 | 1 | 1
0 | 1 | 1 | 0 | 1
1 | 1 | 0 | 0 | 0

Question 37

What logic function allows you to combine the outputs of neuron 1 and neuron 2 to obtain the desired XOR at the output of neuron 3?

Answer 37

In the first case an AND-operation, in the second example the OR function.