Lecture 6 - Linear Models Flashcards
What is a linear model?
f(x) = w1x1 + w2x2… + b, where x are features, w are the coefficients and b is intercept term
What is usually the error measure for regression line?
Mean square error in y-direction, meaning that the sum of squared error in y-direction is minimized.
What is the problem with linear regression?
It is prone to overfitting in high dimensions, and it is sensitive to outliers
What is ridge regression?
It is a regularized least-squares, meaning that it adds regularization term to the mean squared error that penalizes too complex models.
What are the advantages of ridge regression?
Provides more robust solution than least-squares especially for high-dimensional data.
What is the computational complexity for ridge regression?
O(d^3 + d^2n), so feasible if dimensionality d is not too large
What is Lasso?
Lasso is basically the ridge regression, but instead of having w^2 in the regularization, it uses absolute value of w.
What is elastic net?
Combination of ridge regression and lasso
What is support vector regression?
Uses ridges regularizer, but only penalizes error that are higher than tolerance
What is a linear classifier?
A linear model that defines a hyperplane that splits data into 2 classes.
What is a 0-1 loss function?
If y - pred(y) < 0, then 1 (FALSE) else 0 (TRUE). This means that there is a loss of 1 when the prediction is false.
What is squared loss?
(pred(y) - y)^2
How does support vector machine work? What is Hinge loss?
It maximises the margin separating the classes.
Hinge loss: max(0, 1-y*pred(y))
What is sigmoid function?
1/(1+e^-z), always gives values between 0 and 1.
What is logistic regression?
Logistic regression tries to minimize the logarithm of the likelihood of training data, and gives predictions scaled between 0 and 1.
