Fully Connected Networks Flashcards
What is the purpose of the synapse in human memory?
The synapse is a ‘gap’ between neurons. When neurons want to communicate over this gap, they send chemicals called neurotransmitters into the synapse.
When it successfully communicates, that gap gets stronger and more accurate, allowing faster recall.
What is the purpose of the input layer in deep learning?
The input layer takes a set of inputs and passes them through several weighted edges into a ‘hidden’ layer. For example, in the case of an image, the input layer is each individual pixel in that image.
What is the purpose of the hidden layer in deep learning?
The hidden layers in a DL network are a black box - they are neuron layers hidden from the user in-between the input and output.
Weighted edges carry data from one neuron to another, performing transformations on those values non-linearly using activation functions.
What is the purpose of the output layer in deep learning?
The output layer defines the different classes of output. In the case of a Boolean output, we would have 1 or 2 neurons, representing ‘true’ or ‘false’.
What is the purpose of a loss function in deep learning?
A loss function is a function that measures how badly the AI system did at predicting a labelled output using some ‘badness’ score, comparing predicted to actual.
What is the purpose of the optimizer in deep learning?
The optimizer takes the loss value produced by our loss function and adjusts the weights between our neurons to allow our network to ‘learn’.
What is the purpose of activation functions in deep learning?
They introduce non-linearity into the network, transforming the summed weighted input to a node to an output that can be passed on.
What is backpropagation?
Backpropagation is a technique to allow a network to learn by mathematically propagating the value a loss function produces backwards through the network, adjusting weight values.
How is gradient descent used to optimise our network?
Gradient descent is an algorithm used to find a local minimum of a differentiable function.
If we can find weighted values such that they produce a local minimum for our loss values (a point where the loss is exactly zero), we have a perfect model.
What would happen if we didn’t use activation functions?
Our network would essentially reduce to a linear regression, where the relationships in our data would be all part of one straight line.
What are the steps to performing backpropagation on a neural network?
Firstly, draw a batch of training samples and corresponding targets. Then, run the network on those values to obtain a set of predictions.
Compute the loss of the network (the mismatch between the predictions and the targets), and update the weights of the network in such a way that we slightly reduce the loss on that batch.
What happens if we leave our step size too high?
Think of it like a wheel rolling down a hill - if it goes too fast, it will jump straight past the minima, perhaps finding a less optimal point. If it goes too slow, it will never reach it. We need a perfect step size that will allow us to hit the minima without jumping it.
What are hyperparameters?
Hyperparameters are global parameters that do not change as the network is running, but can also be optimised to help the network reach an optimised state. Step size (or learning rate) is a hyperparameter.
What is an epoch?
An epoch is a hyperparameter - a single pass-through of the network by the training samples. The more epochs we set, the more times the data will be trained on the training set.
What is batch size?
Batch size is the partition size of training data to be passed through the network at any one time. If we have 100 bits of data, and our batch size is 10, for each epoch, we will send through 10 batches.
