Lecture 3 Flashcards
What is invariance in machine learning?
When some change is made to a data point, the target output is not changed. Therefore the network output should be the same.
Invariance in machine learning is a property of a model that ensures its output remains the same when the input is transformed in a way that doesn’t change the task the model was designed for.
In what contexts is it important to consider invariance?
- Image recognition (classification - does it contain a red square, it doesn’t matter where the red square is in space)
- Text recognition (scaling / enlarging shouldn’t change the answer)
- Talking (invariance under speed - talking at a different rate shouldn’t change the meaning of the sentence)
- Computational chemistry (asking if a drug can be used to treat disease, should get the same answer regardless of the position in space)
What types of invariance are there?
- Translational
- Rotational
- Reflection
- Scaling
- Swapping (permutation)
What ways can we ensure that a model has the necessary invariances?
[Not clever but easy]
- Augment the training set with transformed examples
- Extract features that are also invariant
- Build the invariance into the network structure (this leads to CNNs)
[Clever but hard]
Give an example of augmenting the training set with transformed examples.
Rotating/tilting images
For invariance, if we apply some transformation T to the inputs xi, what happens to the output ti?
The output ti is unchanged.
How can the network learn this invariance?
If we augment the dataset. D –> D’
The original dataset contains the original input and output, in addition to the transformed input and output. In this way, we have doubled the size of the dataset. We now have a bigger dataset to train the model.
What are the advantages and disadvantages of augmenting the dataset?
Advantage
- Very straightforward to implement
- Can produce a score ???
Disadvantage
- Makes the training set much bigger
- It will only learn the invariance approximately (unless you have infinite data)
How can we enforce invariance exactly?
If the features input into the network are invariant to the transformation.
Harder, but more robust (it guarantees symmetry)
What is an example of invariant features that can be used?
We explored the example of energy of two atoms. Each has a position in space represented by three coordinates and energy as an output.
We could naively use the coordinates as features. However, these are not invariant to different operations. Energy is invariant to translation, rotation and permutation of atoms (ie swapping atom 1 for atom 2).
We find that we can only depend on the norm relative vector of two atoms. This also reduces the number of features from 6 to 1. A single feature now contains all of the information we need and is invariant to all of the required transformations.
What does symmetry refer to?
In machine learning, “symmetry” refers to the concept of a data pattern or transformation where an object remains unchanged under certain operations, like rotation, translation, or reflection, which allows models to learn more efficiently by leveraging these invariances in the data
Describe the difference between symmetry and invariance.
In machine learning, “symmetry” refers to a property of the data itself, where certain transformations can be applied without changing its underlying structure, while “invariance” describes a model’s ability to produce the same output regardless of those transformations applied to the input data; essentially, symmetry is a characteristic of the data, while invariance is a desired feature of a machine learning model that leverages that data symmetry.
What kind of neural network architecture can we build that gives invariant predictions?
Convolutional neural networks (CNNs) account for translational invariance in the inputs.
In a deep NN, the outputs from one layer can be thought of as the features input into the next layer - they have this modularity.
How are images processed in the brain?
Hierarchically
We start off by collecting low-level information on small scales and build our way up, passing information up the hierarchy. It is transformed into higher-level (and larger-scale) information.
This is similar to a deep neural network.
What is the problem when using a fully-connected deep neural network for processing images?
It would require a lot of weights.
Eg camera pictures can have 2M pixels, each with a corresponding RGB value, leading to 6M pieces of information. If we had 2000 nodes in the first hidden layer, there would be around 4B weights.
High-quality image recognition with a fully-connected deep neural network is unfeasible. We can help matters out by noting that these networks do not take into account spatial correlation.
Why should spatial correlation be considered in image processing?
Parts of an image that are closer together are more likely to be similar to each other than those that are far apart. We want to combine information from input features that are correlated with each other spatially.
What do convolutional neural networks (CNNs) do and how do they achieve this?
CNNs mimic the processing of images in the human brain.
They consist of feature learning and classification. Feature learning involves two new kinds of layers, convolutional and pooling layers. From this, we get a much smaller output set of convolutions which represent the image and these are fed into the standard fully connected layers for classification.
What are the two different layers of a CNN and what do they do?
- Convolutional layers - search for visual elements in groups of spatially correlated pixels
- Pooling layers - combine information from nearby pixels to create a smaller image
In this order usually
What does a convolutional unit / kernel do?
Looks at small groups of input features (pixels). It has a receptive field, which refers to the specific area of an input image that a single neuron in a convolutional layer of a neural network can “see” and use to calculate its output
They scan over the whole image, combining the inputs in their receptive fields
What is the output of a convolutional unit?
A smaller image
What is the size of a convolutional unit?
(n x m)
How many input units it is connected to
What is the stride s?
How often the convolutional unit observes information.
How many convolutional units in a layer can you have?
Many, each one scans for “something different” and produces a separate output image.
What letter is used to represent the convolution kernel?
W - weights of the convolutional kernel
How do you determine the output of a convolutional unit?
- Determine the size of the output
- Multiply and sum: “scan” the unit over the input image - multiply the weights and the sum. It is NOT matrix multiplication, it is element-wise. Multiplying the convolutional unit by a portion of the image matrix each time.
- Add a bias: add/subtract the bias from each element. This is an extra weight.
- Apply the activation function
What do convolutional units act as?
Feature detectors
How is the bias represented?
wo
What are the weights of a convolutional unit tuned to detect horizontal lines (3x3)?
0 0 0
1 1 1
0 0 0
If a convolutional unit is tuned to detect horizontal lines, what are the expected outputs (following bias and activation function)?
If a horizontal line is present, there will be an output of 1 and the unit will “fire” due to this input. The output is a 1x1 image with element 1. The unit will only have an output when presented with a horizontal image.
If a horizontal line is not present, there will be an output of 0 and the unit will not “fire” due to the input.
When we apply a convolutional unit to a portion of the image, W * X1, what are we doing?
Convoluting the image with the CU
How can you use multiple convolutional units to detect more complex patterns?
Eg want to detect a cross
We can have two convolutional units, one for a diagonal line sloping down and one for a diagonal line sloping upwards. Where both fire, there is an intersection of the lines and a cross is present.
How do we decide which convolutional units to use / which features are important?
We as humans do not, the network learns what the important features are in the images presented to it - ie it will learn convolutional units based on features.
How are the features that are detected by given convolutional units determined?
By the weights, which are set during learning.
What is an advantage of CNNs over fully-connected networks?
The memory cost is smaller, as the number of weights is very small compared to fully-connected network.
Invariance - all parts are treated equally, so if part of the image is translated, it will still be recognised in the same way.
The convolutional units tell you what the ML model has identified as important - what kinds of features were detected and were important.
What was one of the first CNNs used for handwriting?
LeNet-5
Convolutional units scan over the entire image - what does this mean for parts of the image?
All parts of the input image are treated equally.
If part of the image is translated, it will still be recognised the same way (invariance).
What can a single convolutional layer detect and what is the output?
A single convolutional layer detects simple features from the inputs. The outputs of convolutional units are smaller images.
What happens to the output of single convolutional layers?
These outputs can be fed into more convolutional layers, which put simple features together to make more complex ones.
Each convolutional unit scans over the image to produce a new image, each looking for a different (basic) feature. We can stack up these images to produce more complex ones.
Eg simple straight lines - build up to corners
Why can we pool information?
If the input data is correlated spatially, some of the information is redundant. Nearby points will be very similar to each other. We don’t need to keep all of this data and we can compress the image without losing a lot of information.
We combine (pool) sets of nearby pixels into one.
Describe the pooling unit.
The pooling unit has no weights - it combines pixels (units) in a predetermined way to make a smaller image. For example, using max pooling the pixel of a subset with maximum value of is used to represent the subset. In this way, a 2x2 square is combined into a single value. We could also use average pooling. NO WEIGHTS INVOLVED IN THESE OPERATIONS.
Describe the structure of a CNN.
CNNs are comprised of several layers, which may be fully-connected, convolutional or pooling.
Typically feature learning section - consisting of convolutional and pooling layers. In a way, this could be considered pre-processing. At the end of this, we have a stack of images out of convoluting and pooling. There is much smaller feature space (a much lower dimensions of features).
Followed by fully-connected layer used for classification. We get probabilities from this which determine the prediction.
What can be said of the dimensionality of CNNs?
High dimensional input –> much lower dimensional output
What are some examples of what CNNs are used for?
- Mostly image recognition
- Handwriting detection - classifying written letters/numbers
- Facial recognition - user authentication
- Object detection - eg detection and classification of obstacles by self-driving cars
- Computational chemistry - drug discovery, atomistic simulation
- Board games - AlphaGo (2014)
Describe AlexNet
One of the most modern CNNs, using GPUs in training. It popularised CNNs for image classification, had around 15% error.
The first layer detected edges, lines and gradients of colour. Layers resided on different GPUs (different GPUs looking for different features) - this emerged it wasn’t a constraint.
The second layer than detects more complex features (corners). The complexity of the features detected builds up with each layer.
Dog breeds.
What improved the error of AlexNet?
GoogLeNet - 22 layers and achieved around 6^ error, which is comparable to human performance. Utilised inception nodes.
What does the use of lots of deep layers mean?
Lots of weights which can lead to overfitting.
What is a CNN used for chemical systems?
SchNet
