Perceptron & Backpropagation Flashcards
How does the lecturer compare a brain to a computer?
Brains compute actions from perception
=> hidden states(info stuck in brain that influences what you’re going to do): goals, plans
=> learning: Brain is flexible, learning going on
▪outcome dependent (reinforcement/supervised; ‘if i go to bed at the right time i’ll get a good nights sleep’)
▪outcome independent (unsupervised/self-supervised)
Describe how spiking neurons are modelled
early models: spikes (McCulloch Pitts)
An input affects the firing rate which can be an analogue value between 0 and 200 Hz
h = hidden layer, something that you can’t observe; A hidden layer is located between the input and output of the algorithm, in which the function applies weights to the inputs and directs them through an activation function as the output.
n neurons in hidden layer, n in next layer –> total of n^2 layers
What functions can the transfer function take?
Linear, hyperbolic tangent etc.
What is the purpose of the activation and transfer function?
Activation functions work on some threshold value and once that value is crossed the signal is triggered, While Transfer function is used to translate input to output signals.
Give a very simplified generalised model of an artificial neural network
O = f(x1, x2)
(Output is a function of input variables)
What is meant by a deep neural network?
A network with many hidden layers
Name five identification processes in a deep neural network
- Identify pixel values
- Identify edges
- Identify combinations of edges
- Identify features
- Identify combinations of features
Give an example of objective functions irl
ultimate objective function = survive and procreate
but its not just that
How are these identification process compared to physiology?
some neurons become sensitive to edges
some neurons are sensitive to parts of the face
Describe the relationship between parameters and fit
Too few parameters is insufficient and does not capture the data well enough, too many parameters fits the data too well, overfits and does not describe the underlying generalisable relationship. Taken as an analogy of a line on a graph, the lines could perfectly outline the shape in an image of an elephant but this would not generalise to other images of elephants
What is a loss function?
A loss function is a function that compares the target and predicted output values; measures how well the neural network models the training data. When training, we aim to minimise this loss between the predicted and target outputs.
What does the architecture of a network concern?
How does a certain architecture with certain rules achieve certain objective functions?
Where does neurological preprocessing take place? How are the parameters updated?
Retina does fantastic preprocessing and is fixed
What is visual processing mostly based on?
Visual processing is mostly based on contrast
What are the learning rules of an algorithm; what methods do they employ?
They use a gradient and a learning update rule. The loss function is a method of evaluating how well your machine learning algorithm models your featured data set
As a summary what does neural learning combine?
Architecture, loss function(s), learning rules
Give two examples of these neural learning rules
Cortical columns (architecture), plasticity rules
Learning rules get neural networks to do useful stuff. Give some examples of useful stuff
=> Identify digits in images (MNIST)
=> Translate Chinese-to-Dutch
=> Recommend movies based on past ratings (Netflix prize)
=> Modeling cognitive processes:
Attention, perception, etc.
What is the function of learning algorithms?
Finding suitable set of parameters (weights)
Describe the maths behind a simple neuron in a neural network
𝑠 = 𝑏 +∑𝑤i𝑥i
s = activation)
𝑤i are the weights and 𝑏 is the bias in the network.
𝑓 is termed as the activation function.
𝑓(𝑠) is the output of the neural network.
𝑓 (𝑠) = 1 𝑠 > 0; 0 s ≤ 0
Describe how this neuron works
This model can work on non-boolean values where each input connection gets associated with a weight. Here the function calculates the weighted sum and based on the threshold value provided, it gives a binary output.
Describe the initial layer of a perceptron with a 28x28 pixel image
With a given input image of 28x28 pixels, a perceptron can attempt to output which number is shown in the image. This translates to 784 ‘neurons’ each with their own activation, which can roughly correspond to how ‘bright’ it is in the image below ranging from black at 0 to white at 1.
What would the outcome layer consist of?
The last column, the output layer, comprises the nine possible numbers the network could categorise the input into. Each of these ‘neurons’ also have an activation relating to how likely the number is given the input.
What do the layers in between consist of? escribe how they work
The layers between these two are called hidden layers. This network can have e.g two layers of 16 neurons each but that is somewhat of an arbitrary choice here. The activations of one layer (e.g input layer, hidden layer) determine the activations in the next layer (e.g hidden layer, output layer). This network has already been trained to learn digits. This means that if you feed it an image with different activations in each pixel, this causes a very specific pattern of activation in the next layer, which gives some pattern to the one after it, which gives a specific pattern to the output layer. The neuron with the most activation is then selected for what the image represents.