Deep Convolutional Neural Networks

Question 1

What are some popular methods of Deep Learning?

Answer 1

Convolutional Neural Networks

Reinforcement Learning Networks

Generative Neural Networks

Recurrent Neural Networks

Graph Neural Networks

Question 2

How is a Convolutional Neural Network structured?

Answer 2

Neurons are arranged in 3D, where each neuron is only connected by a small region of previous layers

Features are learned in a hierarchical structure, from low level features to high level features

Question 3

What are the basic components for Convolutional Neural Networks?

Answer 3

Convolution Operator
Pooling
Activation Function
Fully connected layer
Loss Functions
Optimisation methods

Question 4

What is the Convolution Operator in CNNs?

Answer 4

It is a ‘panel’ which has pre-meditated values that is placed over a pixel. The ‘panel’ specifies what the output pixel is, by multiplying each pixel’s value with the value that the ‘panel’ dictates. For example:
0 1 2 is overlayed with a panel that looks like this: 4 2 1. The result for that middle pixel is (0 * 4) + (2 * 1) + (1 * 2) = 4.

Question 5

What steps do you need to take to be able to work out the Total Learnable Parameters in CNNs?

Answer 5

1. Take the amount of parameters per filter e.g. RGB = 3 parameters per filter
2. Multiply step 1’s result with the size of the filter e.g. 3 x 3 filter * 3 (RGB channel)
3. Take step 2’s result, and multiply it by the number of filters
4. Work out the total bias parameters, by multiplying the amount of filters by the amount of bias per filter
5. Add step 3 and 4’s results together to get the total amount of learnable parameters

Question 6

What is the main purpose of adding activation functions to anything in ML?

Answer 6

Adding activation functions introduces non-linearity to the network, so that they can learn a non-zero intercept and understand more complex relationships in the data.

Question 7

What are the main features surrounding the Sigmoid function in CNNs?

Answer 7

It can convert the input range to [0, 1]
It saturates and kills gradients with extremely small gradients at regions of 0 and 1

Question 8

What are the main features surrounding the Tanh function in CNNs?

Answer 8

It converts the data into a zero-centered range [-1, 1]
It saturates and kills gradients with extremely small gradients at regions of 0 and 1

Question 9

What are the main features surrounding the ReLU function in CNNs?

Answer 9

It doesn’t saturate and converges faster
Simple to calculate
Some neurons can be dead with a negative input (so this is better with a smaller learning rate)

Question 10

What are the main features of Leaky ReLU in CNNs?

Answer 10

It overcomes the ‘dying neuron’ problem found in the other activation functions
The performance is not consistent

Question 11

Why are CNNs better than MLPs when using image data?

Answer 11

They are more efficient for image data due to the use of:
- Shared weights
- Exploit spatial hierarchies
- Requires fewer parameters
- Translation invariant

Question 12

What does Pooling do in CNNs?

Answer 12

Pooling takes a Feature Map i.e. a matrix of pixels, and downsamples it i.e. reduces its size, whilst also preserving their depth

Question 13

What two types of Pooling are there, and how do they work in CNNs?

Answer 13

Max pooling - Takes the maximum value from each region/window of the feature map

Average Pooling - Computes the average of values in each region/window of the feature map.

Question 14

What does the Fully Connected Layer effectively become for a CNN?

Answer 14

The Fully Connected Layer fundamentally becomes an MLP, where all nodes in a layer are connected to all nodes in the next layer, etc…

Question 15

What is the equation for the loss function Cross Entropy (Classification)?

Answer 15

L = - (N sigma n = 1) (C sigma k = 1) (y_kn * log (yhat_kn))
In other words:
For each data sample and for each class, the true label is multiplied by the log of the prediction.

Question 16

What optimisation methods are used on CNNs?

Answer 16

Gradient Descent
Gradient Descent with momentum
Adam
Back propagation with chain rule to optimise weight W and bias B

Question 17

What is the key feature of Gradient Descent with Momentum in optimisation?

Answer 17

Avoid the local minimum by effectively ‘sliding down’ the curve, then carrying ‘the momentum’ slightly up ‘the hill’ so as to break out of local optima.

Question 18

What is the key feature of RMSProp in Optimisation?

Answer 18

It contains an adaptive learning rate for each parameter

Question 19

What is a short explanation of Adam in regards to Optimisation?

Answer 19

It combines both RMSProp and Gradient Descent with Momentum, and is highly popular in Deep Learning

Question 20

What three optimisation strategies are commonly used with Big Data?

Answer 20

Intrinsic parameters are updated based on the averaged loss value after seeing all the training examples - Too slow to compute when there’s too many training examples

Stochastic Gradient Descent - Randomly selects one training example for gradient calculation

Mini Batch Gradient Descent - Randomly select a batch of training examples for Gradient Calculation

Question 21

What are some major applications of Convolutional Neural Networks?

Answer 21

Object Classification
Facial Recognition
Real-time object detection

Question 22

What methods are employed in CNNs to reduce overfitting?

Answer 22

Data Augmentation
Drop Out
Transfer Learning

Question 23

How does Data Augmentation work in regards to CNNs?

Answer 23

Data augmentation increases the number of training samples by varying the original image’s geometry and appearance

Question 24

When is Data Augmentation performed?

Answer 24

It is normally performed during training, but not beforehand

Question 25

How does Drop Out work in regards to CNNs?

Answer 25

It randomly disables certain neurons during the training process
This is typically within a fully connected layer with a drop out rate, such as 20%.

Question 26

How does Transfer Learning work in regards to CNNs?

Answer 26

It trains the CNN using a large dataset
Afterwards, it freezes the parameters of the first few layers, and only retrains the high-level feature layers or fully connected layers for a new application with a small number of training samples