Activation Functions Flashcards
What step occurs after the weighted summation?
The activation function
What does the activation function do?
provides an ouput number
Why must the activation function be non-linear?
Because you can’t transform the way data is represented with a linear function
What does the output from the first hidden layer become?
the new input for the second hidden layer
This process is repeated until the final output of the neural network is achieved
What are the four primary types of activation function?
- Sigmoid function
- Tanh function
- ReLU function
- Leaky ReLU function
Sigmoid function
-No constraint on the input, can be positive or negative (x axis)
-Output can be bounded between 0 and 1(y axis)
-Use this if you want the output between 0 and 1 (i.e. a probability), such as image identification
Tanh function
-no constraint on input
-output bounded by -1 and 1
ReLU function
-Only when the weighted sum (which is the x axis) becomes larger than 0 will this outcome occur - known as a threshold function
*Leaky ReLU function
-When x increases the y increases a little bit
What does a neural network learn to do in the hidden layers?
to approximate any transformation function
transformation function
the combination of the activation functions the neural network is doing
What function can you use when you don’t want a negative ouput?
sigmoid or ReLU function
e.g. calculating the cost of a house
Which function should you use for the stock market?
tanh function
What function would you use if asking AI to identify an object?
sigmoid function because constrained between 0 and 1
sigmoid function conceptual explanation
-When your weighted sum becomes large, the y value approaches 1.
-The larger the weighted sum, the more confident the activation function becomes.
-If the weighted sum becomes very small, the output will be close to 0 so the probability is very low.
-If weighted sum adds to 0, then its 50/50 - it has no idea
What happens when the e value becomes very large?
the probability is driven down
