Lecture 6

Question 1

When is a model not a deep learning one?

Answer 1

When the input unit is directly connected to the output, without any layers (of transformation) between them.

Question 2

What are the four common activation functions?

Answer 2

1. Sigmoid
2. ReLU
3. Softplus
4. Tanh

Question 3

What does the Sigmoid activation function do?

Answer 3

Ensures that the output is bound between 0 and 1.

Question 4

What does the ReLU activation function do?

Answer 4

Enures the output is bound between 0 and max, so there is no negative output.

Question 5

What are the problems with the ReLU function?

Answer 5

Not continuous, and calculating derivative is challenging and slow.

Question 6

What does the Softplus activation function do?

Answer 6

It is a smooth, continuous version of ReLU.

When negative values get close to zero, the function begins smoothly ascending.

Question 7

What does the Tanh activation function do?

Answer 7

Ensures the output is bound between -1 and 1.

Question 8

When is Tanh (activation function) useful?

Answer 8

When you want your network to be insensitive to numbers farther from zero, and more sensitive to numbers closer to zero.

Has a really good decision factor for when the model knows how close it is to 0.

Question 9

Hidden layer

Answer 9

Any layer in a Neural Network wherein units are not directly connected to the outputs

Question 10

Uses of Gradient Descent

Answer 10

• Learning the weights in computation graphs
• for calculating weights leading into units in the output layer
• for calculating weights leading into units in the hidden layers

Question 11

What is the Loss function?

Answer 11

(y - ŷ)²

Question 12

Vanishing Gradient

Answer 12

When the “error” signals are extinguished altogether as they are back-propogated through a deep network with many layers

Question 13

How does a Vanishing Gradient occur?

Answer 13

• The gradient is very close to zero or exactly zero, so changing the weights leading into unit j has a negligible effect on its output.
• Floating point precision limitations of the hardware

Question 14

How do you solve the Vanishing Gradient problem?

Answer 14

By renormalizing the vector of gradients as you propogate it back through the network, so you ‘boost’ the gradients.

Question 15

What are the implications of a Vanishing Gradient?

Answer 15

The network will eventually freeze and no longer improve, even if there are more layers that need to have the loss back-propagated.

Question 16

Gradient

Answer 16

Percentage of loss every node receives and wants to feed backward

Question 17

Softmax

Answer 17

Softmax function takes a vector of values as input, and outputs a vector of non-negative numbers that sum to 1