L7: Non-Linear Regression Flashcards
After working your way through this module, you should be able to: - Explain the limits and constraints of nonlinear methods - Implement and assess different nonlinear models, such as splines, additive models, etc.
What is the most basic of non-linear models?
The polynomial regression model: we adjust the function of x so that it includes higher degrees of the variables:
What is the method behind step functions?
Are these non-linear functions?
Yes this is a non-linear function.
The step function will split the data into certain cut-points along the predictor axis (usually with cut-points that are based off of prior knowledge)
What are Piecewise Polynomials?
This is a spline model type.
It involves different polynomials in regions defined by the ‘knots’. In such a way we have different polynomial functions for X in different parts of its domain.
Name the spline type
Continuous 1st and 2nd derivatives
Cubic Spline
A piecewise cubic polynomial with continuous derivatives up to order 2 at each knot
Name the spline type:
Not continuous 1st and 2nd derivatives
Polynomial shape
Piecewise Cubic
Name the spline type:
Not continuous 1st and 2nd derivatives
Polynomial shape with sudden change of slop at the knot
Continuous Piecewise Cubic
What happens at the boundaries of natural cubic splines?
The models become highly variable.
- There is less data at the boundaries
- Less certainty here
We can add constraints at the ends, e.g. to constrain the outer knots 2nd derivatives to be zero
A cubic splint with K knots has how many degrees of freedom? How many parameters?
It will have K + 4 parameters / degrees of freedom
A natural spline with K knots has how many degrees of freedom?
K degrees of freedom
Explain the following function. What is it? What is the LHS, RHS?
This is the smooth spline function.
LHS: This is the Residual Sum of Squares, we minimise this to reduce the error of the model.
RHS: this is the smoothness penalty. This is a penalty that penalises the model if it is too convoluted. (Essentially, the area of the 2nd derivative graph)
Hence, if the area under the graph is large (high amount of change) then the model will be penalised and coerced into being smooth
What is Local Regression?
Local regression is where localities of data points are given separate functions depending on the position of the neighbourhood.
- Gather the fraction of training points closest to xo
- Assign weight to each point, within certain distance, far = 0 , close = high
- Fit weighted least squares regression of the y based on the neighbourhood by finding B that minimises the local error
What is the Generalised Additive Model?
The GAM is a general framework for extending a standard linear model by allowing non-linear functions of each of the variables
We go from y = B0 + B1x1 + B2x2 …
to
y = B0 + f1(x1) + f2(x2) …
Such that the f(x) can be a non-linear function of the variable x
