Lecture 6 - Linear Models Flashcards
What kind of model are Linear models
Geometric models
What are linear functions
f(ax1+bx2)=af(x1)+bf(x2)
What do geometric concepts such as lines and planes do?
- impose structure
- Represent similarity between points
List 4 properties of linear models
- Simple
- Parametric
- Stable
- Prone to underfitting
What is Parametric
We know in advance what are the parameters that need to be learned
What is Uni-variate linear regression
in uni-variate linear regression f(xi)=axi+b, where b is referred to as intercept, a is called regression coefficient.
What is linear regression
Linear regression is about finding the parameters a and b such that sum of residuals Error f(x1)-^f(x1) is minimised
how to find the parameters a and b
Goal is minimizing the sum of squared residuals. We set the partial derivatives to 0 and solve for a and b
Linear regression is susceptible to ____
outliers
Name 2 ways how we can evaluate performance of regression
- Root mean squared error
- R^2 or the coefficient of determination
- RSS
- TSS
What can we do about the outliers? Name 2 solutions
- Using the ordinary least squares method
Train the model
Filter out the noisy pionts absed on the residuals plot
Retrain the model - Using the total least squares method
Total least squares: generalises the least squares method to the situation that both x and y values are noisy.
If we have very few data points during training, the linear model might not be representative of the test data, why?
Low residuals on training data and high residuals on test data lead to overfitting
What is regularisation
Regularisation is a general method to avoid overfitting by applying additional constrains to the weight vector w. A common approach is to make sure the weights are, on averagem small in magnitude: this is referred to as shrinkage.
what is the regularised regression expression
w=argmin(y-Xw)T(y-Xw)
What is a perceptron in linear classification
A linear classifier that will achieve perfect separation on linearly seperable data is the perceptron. The perceptron iterates over the training set, updating the weight vector every time it encounters an incorrectly classified example.
