Neural Networks

Question 1

Activation Function

Answer 1

The ‘gate’ between the current neuron and the output for next layer. Typically a sigmoid/ReLU function that takes in all weights and inputs.

Question 2

ReLU

Answer 2

Rectified Linear Unit – an activation function thats a simple step function, either 0 or positive.

Question 3

Feed Forward Neural Network

Answer 3

Simplest form of Neural Network. Information feeds forward. This is in contrast to RNNs where information can occur in loops etc. This has nothing to do with backpropagation.

Question 4

Backpropagation

Answer 4

The training algorithm which goes back from the output layer and uses gradient descent to determine how the weights need to be updated based on how off the output was.

Question 5

Gradient Descent

Answer 5

For each change in the weights of a neuron, which direction has the greatest effect on the cost function. Based on this gradient, it optimizes the cost function.

Question 6

Gradient Checking

Answer 6

Checking the gradient descent algo during backprop against real examples in order to make sure algo is working correctly.