Logistic regression

Question 1

For the binary classification problem, define the log ratio as:

a = log [(p(C1|x) / p(C2|x)]
= log[(p(x|C1)p(C1) / p(x|C2)p(C2)
What is sigmoid(a) = 1/(1 + exp(-a))

Answer 1

sigmoid(a) = p(C1|x)

Question 2

How can we generalize logistic regression to K classes ?

Answer 2

Use softmax,

p(Ck|x) = p(x|Ck)p(Ck) / sum[p(x|Ci)p(Ci)]

Question 3

In the usual binary logistic regression setting, what kind of distribution is the likelihood?

Answer 3

Bernoullli

Question 4

What is the NLL (Negative log likelihood) of the binary logistic regression?

Answer 4

• sum y_i log (mu_i) + (1 - y_i) log ( 1 - mu_i)

Question 5

What is the derivative of the sigmoid, o(z)?

Answer 5

o(z) ( 1 - o(z))

Question 6

What is the gradient of the NLL for binary logistic regression?

Answer 6

(y - my)^T*X

Question 7

Does the NLL of binary logistic regression have a unique global optimum?

Answer 7

Yes, but no closed for solutions exist

Question 8

How can we solve the problem that under Bayesian logistic regression, the integral of the marginal likelihood is intractable?

Answer 8

Use approximations, for example using the laplace approximation by expanding -log p_hat(x) using the second order taylor expansion around the mode, recognizing the mean and covariance matrix in the resulting normal distrubution.