Activation Functions

Question 1

What are the advantages of linear activation functions?

Answer 1

You can have multiple outputs rather than just 0 and 1 which are porportionate to the input.

Question 2

Whats a disadvantage of linear activation functions?

Answer 2

You cannot use gradient descent as the gradient of a linear function is a constant irrespective of input. Also combinations of linear functions are also linear, meaning you essentially collapse networks into single nodes.

Question 3

What are the advantages of the sigmoid function ?

1/(1 + e^-x)

Answer 3

• Unlike step sigmoid has smooth curve meaning probabilities of things being in a class can be determined as well.
• Outputs are normalised between zero and 1.

Question 4

What are the disadvantages of the sigmoid function?

1/(1 + e^-x)

Answer 4

• Vanishing gradient problem, For large or small inputs the results will not change that much causing slow learning.

Question 5

What are the advantages of tanh function?

(e^x - e^-x ) / (e^x + e^-x )

Answer 5

• Also S shaped allowing continuous results.
• range between 1 and -1
• Tanh’s gradient can reach a max of 1, whereas sigmoid can only reach 0.25

Question 6

What are the disadvantages of the tanh function?

(e^x - e^-x ) / (e^x + e^-x )

Answer 6

• Vanishing gradient
• computationally expensive due to exponentials

Question 7

How are tanh and sigmoid releated?

Answer 7

tanh(x) = 2 * sigmoid(x) - 1

Question 8

what is the derivative of sigmoid?

Answer 8

sig(x)(1 - sig(x))

Question 9

What is the derivative of tanh?

Answer 9

1- tanh(x)^2

Question 10

What are the advantages of the relu function?

Answer 10

• Simple and easy to compute causing fast convergence
• No vanishing gradient problem

Question 11

What are the disadvantages of the relu function?

Answer 11

• negative inputs always result in 0, this can cause the relu neuron to become inactive known as dying relu.
• Not a zero centred function meaning no derivative at zero exists.

Question 12

What is leaky relu?

Answer 12

Like relu but makes negative inputs smaller rather than 0.

Question 13

What are the advantages of Leaky Relu?

Answer 13

• easy to compute
• no vanishing gradient
• no dying relu

Question 14

What are the disadvantages of Leaky Relu?

Answer 14

not zero centred.

Question 15

What are the advantages of ELU?

Answer 15

• smoother than leaky relu at x=0
• no dying relu
• no vanishing gradient

Question 16

What are the disadvantages of ELU?

Answer 16

more expensive to compute than leaky relu.

Question 17

What is softmax?

Answer 17

takes an array of numbers, and normalises them between 0 and 1 based on their exponential values. Used for multi class classification.