Computer Practicals Flashcards
In the lecture, you saw the following equation:
ai=f(∑ wij aj−bi)
Which aspects of the biological neuron are being modelled by the variables?
wij : Weight, so the strength of the synapses j to neuron i
bi: Bias, a value which determines how quickly a neuron will respond to particular inputs.
f: Transfer function, so the transduction of the incoming input to the output of the neuron.
ai: Output of the neuron i, in firing rate.
Which physiological aspects of real (biological) neurons are missing in this model?
Several aspects of real neurons are missing in this model, because this model is a gross oversimplification. Some examples:
Timing of spikes, there is no time component in this model.
Abundancy of dendrites / axon branches and their spatial arrangement.
Refractory period.
Adaptation, plasticity.
No biomolecular effects such as aging, hormones, disease etc.
A biologically plausible transfer function; a hardlimit transfer function is not possible in real life.
Effects of LTP.
Consider a neuron, which receives input from a single presynaptic neuron. We will now drop the subscripts and call the output of the neuron of interest a, the output of the presynaptic neuron p, and the bias b . consider two settings, namely w=1 and w=−1, and b=0.5 with a hard-limit function (hardlim) to start with.
Describe how output a varies as a function of p (where p is between 0 and 1), for both cases w=1 and w=−1.
w=1: This neuron simply multiplied the input p by 1, thus doesn’t change it. If this reached threshold b=0.5 it fires, i.e. the output a=1
w=−1: This neuron received negative input from the negative w, so this can be seen as inhibitory input. With p between 0 and 1, it will never reach threshold and spike (i.e. never reach a=1)
Assume now that the input gets a larger dynamic range and can vary between -4 and 4.
Describe again how the output a varies as a function of p .
When pp is allowed to vary from -4 to 4:
w=1: As before, the in input pp is multiplied by 1, thus stays the same. Ergo, when p reached threshold b=0.5 the cell will spike, as in a=1.
w=−1 : The input is multiplied by w=−1. In this case, a negative input will be able to reach threshold, i.e. p< −0.5. Other inputs will not reach threshold.
We further assume that we receive a binary signal as input (p is either 0 or 1), however now we can vary the bias, b (-2,2).We again study both cases w=1 and w=−1 . For what ranges of the bias b will our neuron be able to discriminate a binary input (which takes on either 1 or 0) coming from the pre-synpatic cell?
w=1: Here the best classification between 1 and 0 is the middle, i.e. b=0.5. As the weight won’t affect the p value, inputs of 0 will be < b=0.5 and inputs of 1 > b, giving appropriate outputs. If the input will only be 0 or 1, b should be in the range 0≤b<1
w=−1w=−1: When the input is multiplied by −1−1, it’s basically the same but negative, ergo bb should be in the range −1≤b<0.
What is the output of the hardlim function?
0 or 1
Give two alternative functions given in the first assignment
Linear function: purelin(x): f(x) = x
Hyperbolic tangent: tanh(x): f(x) = e^2x - 1 / e^2x + 1
What are the codomains of these two functions?
The codomain of the linear function is (−∞,∞), while for the hyperbolic tangent it is (−1,1).
Describe the graphs given by these two functions
Linear function: Gives a linear graph with a continuous output
Hyperbolic tangent: Gives an S shaped curve between -1 and 1 on y axis
What effect does the bias have on these two graphs
Linear function: Shifts the line along the x axis
Hyperbolic tangent: Shifts the line along the x axis
Does not change the shape of the function, just the amount required for the same output, higher bias means more shifted to the right and more input required for a higher output
Which configuration of transfer function and parameters classifies values of the input pp most strictly into ‘categories’? We refer to transfer functions that make such a strong distinction between two ranges of input values as classifiers.
To make a strong classifier, the hardlimit transfer function would be ideal. The values of b and w will depend on what you want to classify, but the w can’t be 0. Dependent on your input p, your threshold should be of a logical value, i.e. inputs and weight of 1 and a threshold of 500 will not classify.
For instance, as seen in previous questions, classifying binary inputs to binary outputs (a really boring classification), can be readily done with 0≤b<1, and a w=1
Conversely, which configuration is the worst at grouping values of the input in separate categories?
A linear transfer function (like purelin) will not give strict classifications, but a continuous output.
What could be the advantage of a poorly classifying configuration?
This will be an estimator. If you want your output to reflect the input in a continuous, linear fashion, you would need this configuration.
Which transfer function is the most biological one? Explain!
Of our transfer functions the hyperbolic tangend (tanh) function is the most biologically plausible one. These shapes of curves are often seen in biology and psychology
In which input range does the hyperbolic tangend (tanh) transfer function behave as an estimator of the input, and in which range as a classifier?
Estimator range: The middle of the tansig curve is relatively straight, and acts as an estimator, much like a linear transfer function.
Classifier range: The ends of the tansig curve plateau towards 0 and 1, and this acts similar to the hard limit transfer function.
What is the essential mathematical feature of a transfer function aimed at classification?
The feature of non-linearity; it needs a relatively strict distinction between values, such as we see in the hard limit function. This is highly non-linear.
Which aspects of how neurons render the transfer from synaptic input to output can be modelled with a sigmoid function?
- The sigmoid function has a threshold-like point, it has to reach certain input to start responding. 2. Moreover, in the middle, a small increase in input gives a large increase in output (i.e. it has a dynamic range). 3. furthermore, the function also has a maximum, where the output doesn’t increase further.
Which physiological aspects of real neurons can be seen as the counterpart of the bias in computational neurons?
This can be thought of as the resting potential of a neurons, spontaneous input of other neurons on said neuron, biomolecular effects (i.e. hormones etc), I also said LTP
What is meant bu Boolean algebra?
Data with only two possible values (think of a neuron that can only produce output which is either 1 or 0) is handled by a branch of mathematics called Boolean algebra, where the values 1 and 0 are often replaced by the labels true and false.
Boolean algebra comes with a set of basic operations or functions, which take as input boolean values and produce boolean output. Give three examples of these
AND, OR, XOR:
OR function, which given two boolean inputs, returns True if at least one of the inputs is True, or the exclusive or (XOR) function, which returns True only if exactly one of the two inputs is true. AND returns true only if both are true
For what inputs of x and y would activate these three functions?
Can be visualised with a truth table:
X | Y | X and Y | X or Y | X xor Y
0 | 0 | 0 | 0 | 0
1 | 0 | 0 | 1 | 1
0 | 1 | 0 | 1 | 1
1 | 1 | 1 | 1 | 0
How are these 1s and 0s represented in Python and other coding languages?
True
Most programming languages implement a boolean data type, which can be represented by 0 and 1 or by True and False labels. In Python, True and False are built-in constants, and we can ask Python to combine them with Boolean operations:
a = True
b = False
print(a or b)
We can also use the numeric values 1 and 0, as long as we tell Python that we mean boolean and not integer values.
a = bool(1)
b = bool(0)
print(a and b)
In electronics, what is a logic gate?
In electronics, a logic gate is a (conceptual) device which implements a boolean function. We can think of neurons as biological logic gates, in the sense that we can consider our modelled neuron as a system which receives binary inputs (e.g. spike or no spike from its presynpatic neurons) and outputs a binary signal (spike or no spike).
How do we broadly programme a neuron at the beginning of assignment 2?
We begin by writing a very simply Python class which we call neuron. This neuron has two methods, namely __init__, which is used to generate instances of this class, and output, which produces the neuron’s output given the inputs.
Write down the general equation that describes output a as function of the inputs pi for some transfer function f. Then, simplify the equation for the case of a linear transfer function and two inputs p1 and p2 .
General equation:
a = f(∑ wi pi - b)
Simplification:
a = f(∑ wi pi - b)
a = f((w1 p1 + w2 p 2) - b) only two inputs
a = w1 p1 + w2 p 2 - b linear transfer function
Given a neuron with 2 inputs and a hardlim function, computing the output using the expression for the linear transfer function, and whenever it is larger or equal to zero we set it to True, and otherwise to False. The parameters are set to bias=1, weight_1=1, weight_2=-1. Fill out a truth table corresponding to the boolean operation performed by this neuron.
1 | 2 | neuron
0 | 0 | 0
1 | 0 | 0
0 | 1 | 0
1 | 1 | 0
Describe why there is no output for this neuron
Since the second weight is set to a negative value, and the input can only take on a positive value, the output of this neuron can only be a negative value. In the hardlim function, the bias can act as the threshold. Since it is set to 1 and the output < 0 the threshold cannot be reached.
What values of the bias and the weights can the output of our neuron become equivalent to the AND function shown previously? (bias can be between 0 and 1; weights can be between -2 and 2 in steps of 0.5)
Bias = 1
W1 = 1
W2 = 0.5
Bias = 1
W1 = 0.5
W2 = 1
Bias = 1
W1 = 1
W2 = 1
If the weights are any less (e.g both 0.5) then their sum does not exceed 1. If the bias = 0.5 then the weights must be 0.5
Now find parameter values such that the computation performed by the neuron corresponds to the OR function.
Bias = 0, 0.5
W1 = 1
W2 = 1
As long as one input fires, it should surpass the threshold
Now try find parameter values such that the computation performed by the neuron corresponds to the XOR function.
You can’t
Can you show mathematically why this is not possible?
The neuron output is defined as: w1∗p1+w2∗p2=a
For the neuron to apply XOR, it should follow these constraints:
1 | 2 | neuron
0 | 0 | 0
1 | 0 | 1
0 | 1 | 1
1 | 1 | 0
Following conditions (2) and (3) we get:
w1=1w1=1
w2=1w2=1
Which, when substituted in (4) gives:
1∗1+1∗1=0
Thus the system cannot be solved.
In general, substituting any 2 values in all 4 equations shows that this set of equations cannot be true at the same time.
What is meant by the decision boundary?
Another way to think about what the neuron is doing is to plot each combination of input values on a 2-D plane. Then, the parameters of our neuron determine a line on this 2-D plane that will separate points in input space for which the neuron becomes active (responds with a 1) from points where the neuron remains inactive (responds with a 0). This line is known as the decision boundary.
Derive the equation for the line which represents the decision boundary, through expressing p_2 as function of p_1. (this will allow you to visualise the boundary)
Starting with:
a=w1p1+w2p2−b
p1 = w2p2 + w1−b-a
Since the transfer function is a hardlim, it will be activated from zero and as such:
0=w1p1+w2p2−b
b=w1p1+w2p2
b−w1p1=w2p2
(b−w1p1)/w2=p2
What would the decision boundary have to look like for the XOR operation to be carried out? Give a graphical explanation of why XOR cannot be solved by a single linear classifier.
The decision boundary would have to be laid out in a manner that activated the bottom left unit as well as the top right unit, however this is not possible with only one linear classifier as there is no line that could be drawn which could only enclose these. A curved decision boundary is necessary. Not linearly separable.
How was this problem solved?
More neurons: the strength is in the union of many computationally simple elements. You can solve this decision boundary plot with two lines
Translate this decision boundary plot into truth table entries by filling in the Output neuron 1 and Output neuron 2
There are two solutions. The first is to have one neuron compute an OR-function and the other a NOT-AND operation. The final computation by the second layer is an AND operation:
1 | 2 | Output1| Output2 | Output3
0 | 0 | 0 | 1 | 0
1 | 0 | 1 | 1 | 1
0 | 1 | 1 | 1 | 1
1 | 1 | 1 | 0 | 0
The second solution is to have one neuron is only activated by Input 1 (negative weight input 2) and the other only by Input 2 (negative weight input 1). The third neuron can perform an OR operation:
1 | 2 | Output1| Output2 | Output3
0 | 0 | 0 | 0 | 0
1 | 0 | 0 | 1 | 1
0 | 1 | 1 | 0 | 1
1 | 1 | 0 | 0 | 0
What logic function allows you to combine the outputs of neuron 1 and neuron 2 to obtain the desired XOR at the output of neuron 3?
In the first case an AND-operation, in the second example the OR function.
