Neural Networks

Question 1

What activation function did the first neural network use?

Answer 1

Heaviside step function

Question 2

What method is used for optimizing multi layer neural networks

Answer 2

Backpropagation

Question 3

Name some improvements to neural networks the past 30 years

Answer 3

1 ) Better hardware

2) Deeper networks
3) Larger datasetets
4) Other changes, better activation funcitons, different layers…

Question 4

How can we adapt gradient descent to work with very large training sets?

Answer 4

Stochastic gradient descent. (Use a random batch from the training data and update the weights using this batch).

Question 5

How does training step influence the error for:

1) Very high learning rate
2) high learning rate
3) low learning rate

Answer 5

1) The error will increase rapidly
2) The error will decrease rapidly in the begining and then “flatten” out, never reaching the optimum
3) The error will decrease slowly.

Question 6

What is the vanishing gradient problem?

Answer 6

Activation functions like the sigmoid saturate for large/small values of x, meaning the gradient is close to 0.

Question 7

What is the exploding gradient problem?

Answer 7

The gradient suddently increases a lot. The gradient descent algorithm can “jump” far away from the optimal solution.

Question 8

How can we adapt gradient descent to fix the vanishing and exploding gradient problems?

Answer 8

1) We can use adaptiv stepsizes.

2) We can Clip the gradient using thresholding or L2 norm.

Question 9

What is the main advantage of the relu over the sigmoid activation function?

Answer 9

It doesn’t saturate for high values of x.

Question 10

How can we deal with relu saturation for input values below 0?

Answer 10

We can use leaky relu, PRelu, or Elu instead.

Question 11

What types of problems is the mean squared loss function most commonly used for?

Answer 11

Regression.

Question 12

What ouput function do we usually use for the binary classification problem?

Answer 12

Sigmoid (Or softmax with 2 outputs, one for “true” one for “false”…)

Question 13

What output function do we usually use for Multi-class classification problems?

Answer 13

Softmax

Question 14

What loss function do we usually use for classification?

Answer 14

Cross entropy

Question 15

What is data augmentation?

Answer 15

We increase the training set by adding distorted, squeezed, tilted… versions of the original dataset.

Question 16

What assumption do we make when using a CNN?

Answer 16

nearby features (for example pixels) are dependent on eachother.

Question 17

How we make sure that the output of a convolutional layer is the same size as the input.

Answer 17

Zero-pad the input.

Question 18

What happens to the output size if have stride=2 in a convolutional layer? (When the input is zero padded..)

Answer 18

The output size is 1/4 of the input size.

Question 19

How does the kernel in a convo layer look if we use dilation = 2 and size 3?

Answer 19

[X, –, X
–, X, –
X, –, X]

We only “look” at values where the kernel is X.

Question 20

What is pooling in a convolutional network?

Answer 20

It is like downsampling. We usually use max-pool meaning that the maximum value in the local region is selected.

Question 21

What is a residual net?

Answer 21

A residual net has “skip connections” mainly allowing the gradient “skip” layers on backprop to leviate vanishing gradients.

Question 22

Name some famous neural networks

Answer 22

GoogleNet, ResNet, AlexNet, VGG.