Linear and Logistic Regression

Question 1

For OLS linear regression on p random variables, what are the input, outcome, action and hypothesis spaces?

Answer 1

IR^p, IR, IR, {w\cdotx + b}

Question 2

What is the OLS loss function?

Answer 2

SE loss

Question 3

Explain the purpose of the gradient descent algorithm

Answer 3

Lazy algorithm: initialise with some weights and biases, calculate the gradient of R with respect to those, update the weights and biases in the direction of the gradient, continue until some condition is met.

Question 4

Define convexity

Answer 4

A convex function has a chord that is always above itself.

Question 5

For a convex function, what does gradient descent always result in?

Answer 5

convergence to the minimum (providing the step size is small enough)

Question 6

What are the two parameters we must specify before applying gradient descent and what are the potential consequences of setting these too large or too small?

Answer 6

Step size and stopping criteria.

If step size too large, may diverge, too small and may take too long to approach the minimum

Question 7

When can/can’t we use gradient descent?

Answer 7

When our loss is differentiable and our hypothesis space is finite dimensional.

Question 8

Explain mini-batch and stochastic gradient descent and their pros/cons.

Answer 8

Mini-batch: gradient calculated on a random subset of the training data. Stochastic: mini-batch with batch size 1.

Question 9

Explain why feature scaling is important in gradient descent.

Answer 9

If different features have different scales, step size can lead to issues (one step size scale for many scales of weights and biases).

Question 10

For logistic regression on p random variables, what are the input, outcome, action and hypothesis spaces?

Answer 10

Input: IR^p
Output: {0, 1}
Action: (0, 1)
Hypothesis: sigmoid(w.x + b)

Question 11

What is the commonly used loss function for logistic regression?

Answer 11

log loss

Question 12

What is the likelihood function?

Answer 12

Probability of seeing our data given our parameters

Question 13

Explain how the maximisation of the likelihood function is equivalent to minimising the log loss function

Answer 13

Maximising the likelihood is the same as maximising the log of the likelihood, use some log laws and we can minimise the log loss.