Artificial Neural Networks Flashcards
Notes on ANNs that may help with the exam.
What are the three types of layers in a Neural Network, and what do they aim to do?
Input Layer - Connects to the input data (typically features)
Hidden Layer - Performs feature interactions (weighted sums)
Output Layer - In charge of the decision making
How does an MLP network function i.e. what are its components?
Lots of neurons are stacked in multiple layers to help learn a complex relationship.
A set of training examples are needed to train the network
Weights that connect neurons are updated to improve the prediction during training
After training, the network can be used to make predictions of unseen data quickly
What does each neuron contain?
Each neuron contains an activation function.
What four activation functions are typically used in MLPs?
Sigmoid
Tanh
ReLU
Linear
What is the activation function’s equation for Sigmoid?
σ(x) = 1/1+e^(-x)
What problem is a Sigmoid Function used for in MLPs?
Binary Classification
What is the activation function equation for Tanh?
tanh(x)
What problems does a tanh activation function solve in MLPs?
Classification problems - Normalises the data effectively and makes it easier for later layers to learn
Regression problems - If the target variable has positive and negative values, tanh ensures the output can match the range
What is the equation for the activation function ReLU in MLPs?
max(0, x)
What problems does the activation function ReLU solve in MLPs?
Non-Linear problems - Used to model non-linear relationships
Deep Neural Networks - Faster and more efficient training than other activation functions.
What is the activation function equation for Linear in MLPs?
f(x) = x
What problem does the activation function Linear solve in MLPs?
Regression - Primarily used in the output layer when solving regression tasks
Why are activation functions useful to have in MLPs?
They introduce non-linearity, allowing neural networks to learn complex patterns in data. Without them, the network effectively becomes a linear model, regardless of layer amount.
What is the step-by-step process for training an ANN?
- Randomly initialise the weights and bias (W & b)
- Input data and perform forward passing, then generate a prediction
- Calculate the error between the predicted output and the true label
- Backpropagate the error from the output layer back to the input layer by calculating the averaged gradients of the batch for each W and b
- Update all W and b using the gradient (gradient descent)
- Repeat steps 2 - 5, until the output is good enough or W and b can’t be further updated.
What are some hyperparameters of ANNs?
Learning Rate
Activation Function/s
Number of Layers i.e. Depth
Number of Neurons in each layer
