4/5 - Deep Convolutional Neural Networks

Question 1

Deep learning

Answer 1

Learning on a network with more than 3 layers

Question 2

Overfitting in deep learning (parameters)

Answer 2

More parameters increases the risk of overfitting.

Too many parameters and not sufficient data points = overfitting

Question 3

Shape analogy for layers (Abstraction)

Answer 3

Layer 1 could detect a horizontal or vertical line
Layer 2 could then detect a shape
Layer 3 could then detect an item.

Abstraction

Question 4

MNIST

Answer 4

Handwriting data set

Question 5

4 AI uses in images

Answer 5

Image segmentation (item separation)
Image captioning
Question answering (is there a boat in this image?)
Action recognition

Question 5

How do you choose the values in a kernel/filter?

Answer 5

Randomly produced and allow the system to learn the best filter

Question 6

CNN Do you flatten the input image?

Answer 6

No, it stays as 2D

Question 6

2D Convolution Layer (5x5 input and 3x3 kernel example)

Answer 6

5x5 Input for example with a 3x3 kernel.

You slide the 3x3 kernel across the image from the top left, sum them and then set the location as that in a new 3x3 matrix

Question 7

Stride

Answer 7

Number of steps taken when sliding the kernel

Eg slide 1 is one pixel
slide 2 is 2

You move faster through the image.

Question 8

Higher stride does what to output?

Answer 8

Reduces the output

Question 9

Padding

Answer 9

Add zeros all around edge and then scan with kernel

Question 10

Computing the size of an output with convolutional:

Answer 10

(W-F + 2P)/S + 1

W input size
F filter size
S stride
P padding

Question 11

Images have how many channels and why?

Answer 11

3 - RGB

(Where the image is RGB of course)

Question 12

3 Channel input and a 3 channel filter. How many channels is the output?

Answer 12

1

Question 13

To make a 3 channel output of a 3 channel input, we need how many 3 channel filters?

Answer 13

3 filters

Question 14

More filters you have means what for parameters?

Answer 14

More parameters. Potential overfitting if too many filters.

Question 15

Must your number of input channels match the number of filter channels?

Answer 15

YES

Question 16

Stack of convolutional layers example

Answer 16

Input
Filters
Feature Maps
Filters
Feature Maps
Filters

Question 17

ReLU

Answer 17

Rectified Linear Unit

max(0,x)

On graph it’s y=0 below x=0 but otherwise y=x

Question 18

When is ReLU needed?

Answer 18

After every convolutional layer.

Question 19

Why can’t we have two convolutional layers right next to each other?

Answer 19

Matrix Multiplication. Two convolutional filters one after the next would be the same as having just one.

We need non-linearity (ReLU etc, sigmoid and others…)

Question 20

Advantages of ReLU vs tanh/sigmoid

Answer 20

• Faster convergence than tanh
• Easier and faster calculation
• Lower probability of vanishing gradient

Question 21

Disadvantages of ReLU:

Answer 21

• Dying ReLU: when inputs are all or mostly negative, recovery is hard because grad is 0 in neg half

Question 22

Pooling Layer: Max Pooling

Answer 22

Takes a sub-part of the image LIKE CONVOLUTIONAL (using a scan kernel) but takes the highest value as the output (rather than multiplying and adding)

Question 23

Pooling: does it collapse the channels, like convolutional?

Answer 23

No, 3 channels in 3 channels out

Question 24

Advantages of pooling layer

Answer 24

• Improved translational invariance (moving pixels to slightly different locations has little effect)
• Downsampling (reduced resources)

Question 25

Batch Normalisation concept

Answer 25

Network is easier to train if the input is normalised

Question 26

Issue with normalisation

Answer 26

In training, the parameters are chaning which leads to changing of data distribution in different layers

Question 27

Batch normalisation

Input: values of x over a mini-batch B = {X1…m}
Parameters to be learned yotta

Answer 27

mini batch mean
mini batch variance
normalise
scale and shift

Question 28

Fully Connected Layer

Answer 28

Each neuron/unit in the previous layer connects to every unit in the this layer.

Question 29

If you had a 9x9 patch and compared a 4 layer approach to a 1 layer approach (ending at 1x1), how many parameters does each have?

Answer 29

1 has 4 biases (filters)
1 has 4 3 3 = 36
4 + 36 = 40 parameters

2 has 1 bias
2 has 1 9 9 = 81
1+81 = 82 parameters

Approach 2 more likely to overfit

Question 30

2 advantages of using more, small filters

Answer 30

• Fewer parameters: lower chance of overfitting and lower computational cost
• More ReLUs: more non-linearity so higher representation capacity

Question 31

Why does GoogleNet arch use several different filters in parallel? What is the problem with this?

Answer 31

Some filter sizes might be better in certain circumstances than others.

The problem is computational cost

Question 32

How did GoogLeNet solve the computational cost issue?

Answer 32

They use Dimension reduction with bottleneck layers.

The 1x1 convolutional layer

Question 33

Residual Block

Answer 33

Takes the input, bypasses the layers and sums it with the layer outputs.

If you’ve already learned then just ensure that f(x) does not do much to effect the input.
ResNet uses them every 2 layers.

Question 34

Densely connected Networks

Answer 34

Residual blocks but connected ot every future layer output.

Question 35

Pros of Densely connected networks

Answer 35

• Stronger gradient in back propagation so it’s easier to train
• Learn feautres from multiple levels
• Reuse of features (fewer filters but same num of feature maps)

Question 36

Drawback of densely connected network

Answer 36

High memory cost. You must save all intermediate feature maps.

Question 37

Dense Block Networks

Answer 37

Uses smaller blocks of densely connected layers, separated by standard layers.

Question 38

Fine-Tuning data requirements

Answer 38

Lots of data required. You could use a pre-trained network then fine tune

Question 39

Fine tuning with less data

Answer 39

You could freeze all layers other than the last and retrain that with only a few data points.

You could also train a linear classifier.

Question 40

Fine tuning with more data

Answer 40

Freeze less layers and train more layers going backwards.

Question 41

Fine tuning with a LOT data

Answer 41

Freeze no layers and retrain all.

Be careful of overfitting

Question 42

Data augmentation

Answer 42

Add transforms like rotation/translation/zooming/noise etc)

Question 43

Dropout

Answer 43

Blackout some units every time you perform inference

Harder to overfit because network can’t rely on one neuron anymore

Question 44

Early stop

Answer 44

You can stop the network when it starts to overfit.