Probabilistic Discriminative Models

Question 1

What are Logistic Regression models?

Answer 1

These are models directly modeling P(Ck|X) by:
P(C1|x) = σ(w.T * x) [for 2-class classification]
P(Ck|x) = softmax_k(w.T * x) [for k-class classification]

Parameters are initialized randomly and their optimal values are approximated through gradient descent.

Question 2

What is Conditional likelihood?

Answer 2

P(T|X, w) = Π P(ti|Xi)

Question 3

What is the parameter update rule of Newton-Raphson method?

Answer 3

w(new) = w(old) - H^-1 * ∇E(w)
(H is the Hessian of E)
N.B. : If the error function is quadratic, Newton-Raphson finds the solution in one step.

Question 4

What is Iterative Reweighed Least Squares (IRLS)?

Answer 4

w(new) = (X.T * R * X)^-1 * X.T * R * Z, were:
-Z is a n-dimensional vector (e.g. Z = X*w(old) - R^-1 * (y-t))
-R depends on w but is not constant, so we must apply this update equation iteratively.

Question 5

What is the derivative of the sigmoid function?

Answer 5

σ’(x) = (1-σ(x))*σ(x)