Beyond Linearity

Question 1

What are some predictive models that move beyond linearity?

Answer 1

1. Polynomial Functions
2. Step Functions - break X into bins, create an ordered categorical variable
3. Piecewwise Polynomilas - fit a different polynomial function to different bins, instead of just different coefficients in each bin (2).
4. Splines - piecewise polynomials that yield a function that is continous across bins.

Question 2

Things you have to worry about w/splines

Answer 2

you have to pick the number of knots, or bins using cross validation. You could choose the location of the knots, but usually just go w/uniform if you dont know.

Question 3

Splines vs Polynomial

Answer 3

Splines usually better, b/c fit lower order polynomial across bins, whereas polynomial must use a higher order polynomial to achieve same effect.

Question 4

Smoothing Spline

Answer 4

Just like splines, but add a penalty or regularization parameters that awards “smoothness”, uses the 2nd derivative.

Question 5

Local Regression

Answer 5

Regression Done iratievely on local values of X, by weighting locally close values higher and far away values almost close to zero. Results in a sort of “moving average regression”. You have to choose the span, or S which is the number of local data points to consider.

Question 6

Consideration of Local Regession

Answer 6

susceptable to high-dimensionality, because lie KNN, in high dimensional space there may be few local data points. Doesn’t usually work if p >4.

Question 7

Generalized Additive Models

Answer 7

Allows you to fit a separate non linear function to each X.

Question 8

Advantages of GAM

Answer 8

1. Can fit non linear functions to each X
2. Potentially more accurate predictions
3. Model is additive, so each effect of X is determined by holding all other vars constant. This means easy to interpret model resutls.