LDA

Question 1

assumptions of LDA

Answer 1

linear decision boundary (linear discriminant function, or the log odd ratio is a linear function of X)
the class-conditional distribution of X is a multivariate Gaussian with the same covariance matrix

Question 2

discriminant function of LDA

Answer 2

a linear function of X

delta_k(x) = x^T \Sigma^-1 mu_k - 1/2 mu_k^T \Sigma^-1 mu_k + log(pi_k)

Question 3

how to estimate the parameters for X distribution

Answer 3

pi_k = n_k / N
mu_k = avg(x_i) when g_i = k
Sigma = avg_k (Sigma_k) where Sigma_k = Var(X) for X in class k

Question 4

QDA (Quadratic Discriminant Analysis)

Answer 4

covariance matrix Sigma_k are different for differen k, the decision boundary is not linear but quadratic

Question 5

Pros and Cons of LDA (Compared to Logistic Regression)

Answer 5

Pros:
* the estimated parameters have less variance if the assumption about X distribution are correct
* can use information from Xs without Y labels to inform Sigma
* well-defined in the case of perfectly separated data
Cons:
* more assumptions about X distribution
* less robust against outliers in the X space (far from decision boundary)