Introduction Linear models and search

Question 1

Describe a perceptron

Answer 1

Multiple features are multiplied with weights, a bias parameter is added to the resulting sum. It is basically a linear classifier.

Question 2

How can we include a nonlinearity to a perceptron?

Answer 2

Use an activation function like the logistic sigmoid

Question 3

For what is a softmax activation used?

Answer 3

Multiclass classification. All output nodes sum to 1

Question 4

What is stochastic gradient descent?

Answer 4

It is a variant of the gradient descent algorithm, but instead of using the entire dataset to compute the gradient at each iteration, it randomly samples a small subset of the data and computes the gradient on that subset.

Question 5

Can you give a summary of how to train a neural network?

Answer 5

Define a loss function
Work out the gradient with respect to the weights
Use (stochastic) gradient descent to improve the weights

Question 6

Give the 3 backpropogation steps

Answer 6

Work out the derivative of the output wrt its inputs symbolically
Compute the global derivative by multiplying all derivatives