Handout #7 - Advances in Network Architecture

Question 1

Explain what transfer learning is

Answer 1

Transfer learning is reusing knowledge learned from a task to boost performance on a related task.

E.g. Using image recognition of cars on trucks.

Question 2

What’s the need for transfer learning

Answer 2

Training a state of the art CNN network from scratch demands a lot of images/data.

Question 3

Explain why you’d use off-the-shelf networks such as VGG, AlexNet or GoogleNet

Answer 3

These OTS networks have been trained for many different images and classes.

This means we only need to design and test the classification part (the DNN layers)

Question 4

If you don’t have enough data, what can you do with the values of the imported weights.

Answer 4

Freeze the values of the imported weights.

Question 5

Explain the problem with implementing transfer learning

Answer 5

Transfer learning means using the already trained network for a similar set of data.

The data might not be normalised -> if we’ve trained the network on sunny days, it’ll have different values to cloudy days. The data isn’t in the correct value range for the use.

Question 6

What can you do to avoid the problem with transfer learning.

Answer 6

You can do BN or normalisation, which shifts the scales the data according to the training set statistics.

Question 7

Explain BN

Answer 7

BN is a normalisation layer, which shifts the data to the correct range.

The values are distributed after BN is 0 centred and variance 1.

It can achieve higher learning rates -> no need to worry about dropout or intialisation.

Question 8

Explain what GoogLeNet and ResNet do to avoid vanishing gradient.

Answer 8

They avoid a pure sequential network (l -> l -> l) and have layers in parallel.

Question 9

Explain how you can add more data in the model

Answer 9

Data augmentation;

Images -> flipping, rotation, zoom

Test -> replace with synonym, shuffle sentences, delete words

Noise -> add noise, reverb, compression

Question 10

What’s the best optimiser learned so far in the course based on -> (i) vast data (ii) small data

Answer 10

(i) SGD
(ii) ADAM

Question 11

Explain what the learning rate scheduler is:

Answer 11

Periodically raise the learning rate to temporary diverge and allow to hope over hills.