Resampling Methods

Question 1

Validation Set

Answer 1

Split 50/50 Test Training. Most Basic approach

Question 2

LOOCV

Answer 2

Leave One Out Cross Validation
fit the data to n-1 observations, and repeat n times over the data. This is computationally expensive as you are fitting the model n times. However, will always yield consistent results. Also makes more data available to the model.

Question 3

K-fold CV

Answer 3

K-fold Cross Validation. This is an alternative to LOOCV where you divide your data into k groups, and use one group as validation, and train data on k-1 groups. Repeat k times. This is the preferred method, as is more realistic estimate of the error.

Question 4

LOOCV vs. K Fold

Answer 4

K Fold gives a more realistic error as models are not as highly correlated w/eachother. W/LOOCV each model is trained on nearly identical data

Question 5

K-fold CV error vs Real Test Error

Answer 5

KFCV tends to underestimate the Test Error, but still finds the minimum point when trying to determine model flexibility (i.e. order of polynomials, or k in knn).

Question 6

Bootstrap

Answer 6

Where you take repeated samples of your dataset, with n observations, with replacement. Replacement means the same observation can repeat itself several times in the dataset. Used to estimate the standard error (variability) in your model output.