Session 7: Regularised Regression

Question 1

What does the standard statistical approach assume?

Answer 1

A parametric model such as the linear model

𝑓(𝑿𝑖 )= 𝛽0+𝛽1 𝒙1+…+𝛽𝑝 𝒙𝑝

Question 2

If we hypothesize a linear model for the response y

𝑦= 𝛽0+𝛽1 𝒙1+…+𝛽𝑝𝒙𝑝 + 𝜺

where 𝜺 =N(0, σ2)

To estimate unknown parameter 𝛽_0,𝛽_1,…𝛽_𝑝 we what?

Answer 2

Ordinary least square (OLS) methods:

We choose 𝛽_0 , 𝛽_1…, 𝛽_𝑝 to minimize the residual sum of squares between observed and predicted responses of the (same) data set.

Statistical inference assuming normal distributing of the error 𝜖 allows constructing confidence intervals around parameters and performing statistical tests.

Question 3

What are problems with the OLS estimator?

Answer 3

There are situations when the OLS estimator does not work well:

When the independent variables are highly correlated- We get unstable, highly variable estimates

The number of parameters relative to the sample size is large - danger of overfitting

We optimise for an unbiased estimator of mean response E(y) but not for predicting new unseen individual cases yi - Often not the optimal model for prediction models, very often the optimal model for explanatory research is not optimal for prediction

Question 4

Describe a model that under-fits

Answer 4

Performs poorly on the training data.

It did not capture the relationship between the predictors X and outcome Y. Performance on test data will be even worse. The model is biased!

No curvilinear relationship thus we assume that it explains least amount of variance in our training data.

Question 5

Describe a model that over-fits

Answer 5

A model overfittedour training data when the model performs well on the training data but poorly on the new test data. This is because the model fitted noise in the training data.

The model memorizes the exact pattern of data it has seen and is unable to generalize to unseen examples, because the pattern will not reappear.

The model variance is high.

We want a model to work well on the future data, not on the training data!

Over-fitted model explains variance best – very small difference between observed and predicted values

Question 6

Describe a balanced model

Answer 6

A model is balanced (“just right”) when it captures the true pattern and therefore predicts well new unseen cases.

Balanced dataset which describes curvilinear relationship should explain variance better than under-fitted model but not over-fitted model

Question 7

Under-fitting is a bigger problem than over-fitting

True or false

Answer 7

True

Question 8

If we assume that there is a relationship between outcome Y (depression score) and at least one of the p independent variables X (clinical and demographic characteristics)

How can we model this?

Answer 8

𝑌=𝑓(𝐗)+𝜖
where 𝑓( ) is an unknown function
and 𝜖 random error with mean 0 and variance 0 and o2

Then the expected mean squared prediction error is:

E(MSE)= (𝑀𝑜𝑑𝑒𝑙 𝐵𝑖𝑎𝑠)2 +𝑀𝑜𝑑𝑒𝑙 𝑉𝑎𝑟𝑖𝑎𝑛𝑐𝑒 + σ2 (noise that we cannot explain)

Want model with smallest prediction error

Question 9

What does the expected mean squared prediction error mean?

E(MSE)= (𝑀𝑜𝑑𝑒𝑙 𝐵𝑖𝑎𝑠)2 +𝑀𝑜𝑑𝑒𝑙 𝑉𝑎𝑟𝑖𝑎𝑛𝑐𝑒 + σ2

Answer 9

Bias is result of misspecifying the model f

Reflects how close the functional form of the model is to the true model

High bias results in underfitting

Model estimation variance is result of using sample to estimate f(x)

Quantifies the dependency of the data points used to build the model.

High variance = small changes in data change model parameter estimates substantial

High variance results in overfitting

o𝟐 is irreducible error even if the model f is correctly specified and estimated

Question 10

How do explanatory and prediction modelling optimize MSE differently?

Answer 10

In explanatory modelling we try to minimize MSE in our training dataset and usually MSE predicted new cases is small than MSE in our training sample

Question 11

How can we improve prediction accuracy?

Answer 11

By reducing the variability of the regression estimates (Model variance) at the cost of increased bias (Model bias) – because OLS is unbiased thereby model bias is 0 and the MSE is caused by model variance plus error

Thus, can confine model that is biased but has smaller variance and this would improve prediction accuracy

The best explanatory model is often different from the best prediction model! – as we optimise a different way

Question 12

If we fitted 90 parameters to a data set of 100 persons, model explains almost 100% of the training data, absolute values of coefficients are too large and we have type I error (too many significant parameters). What does this mean?

Answer 12

Our model overfits the training data and is unlikely to predict well a new random sample

Question 13

Our OLS model is unbiased (with increasing sample size parameters would move towards the true value)

The absolute values of the parameters are too large (many well above 0).

How could we improve the model?

Answer 13

We need to shrink the regression coefficients somehow

• Shrinkage (or regularization) helps preventing linear models from overfitting training data by shrinking the coefficients towards 0.

Question 14

What do shrinkage of regularization methods perform?

Answer 14

Linear regression while regularizing or shrinking the estimated coefficients towards 0.

Question 15

Why does shrinkage help with overfitting?

Answer 15

It introduces bias but may decrease the variance of the estimates. If the latter effect is larger, we would decrease the test error!

Question 16

By sacrificing unbiasedness, we can reduce…

Answer 16

the variance to make the overall MSE lower

Question 17

With regularised methods what do we introduce?

Answer 17

Some bias and on average we are bit away from true parameter but there is very little model variance. All data close to true value and model will predict well. Thus on average closer to true values than ordinary least square method.

Question 18

What did Houwellingen and le Cessie (1990) develop?

Answer 18

Heuristic shrinkage estimate:

𝛾̂=(𝑚𝑜𝑑𝑒𝑙 𝜒2−𝑝)/(𝑚𝑜𝑑𝑒𝑙 𝜒2 )
where
p is the total degree of freedoms of the predictors (number of parameter -1) and
𝜒2 the likelihood ratio statistics for testing the overall effect of all predictors.

For linear model with an intercept of b0 and coefficients 𝛽̂𝑗(j=1,2…p) , the shrunken estimates are easily estimated:

𝑠ℎ𝑟𝑢𝑛𝑘𝑒𝑛 𝛽^𝑗=y^(𝛽̂𝑗) – gamma hat x estimated regression coefficient

𝑠ℎ𝑟𝑢𝑛𝑘𝑒𝑛 𝛽0=(1−𝛾̂ ) 𝑌̅+𝛾̂ (𝛽0)) - Intercept is 1 – shrinkage estimate x mean of all Y’s x mean of all observed outcomes plus shrinkage estimate x intercept

The model with shrunken regression coefficients predict on average better new unseen cases

Question 19

Evaluate Heuristic shrinkage estimate

Answer 19

Works reasonably well in generalized linear models (i.e. Steyerberg et al. 2001)

p is the number of candidate predicators if variable selection is used! – If remove some variables because they are not correlated with outcome, must use variables started with not the final number of variables in model

It would be better to integrate the shrinkage in the model building process.

Find a shrinkage on the parameters that optimizes the prediction of unseen cases
- A general shrinkage procedure are penalized (or regularized) regression methods

Question 20

What is a modern approach to prediction modelling?

Answer 20

Regularized or penalized methods that can be applied to both large data sets (bioinformatics, neuroimaging, wearable data) and small data sets with a large number of variables (RCTs , experimental studies, cohort studies).

Not really new:
Ridge regression: Arthur Hoerl and Robert Kennard (1970)
limited computer power restricted their use

Question 21

What is the basic principle of penalised methods?

Answer 21

To improve prediction accuracy by reducing the variability of the regression estimates at the cost of increased bias (shrinkage)

Question 22

What are advantages to using penalised methods?

Answer 22

1. Also allows automatic variable selection by shrinkage:

As the coefficients of the weaker predictors are shrunk towards zero – removed from regression model

This is very useful for high dimensional data (p&raquo_space; n)

2. Can also effectively deal with with ill-conditioned regression problems

Multi-collinearity (and redundancy - too many variables in dataset and number of variants is close to sample size )

number of variables (p) is close to the sample size (n)

Question 23

1. What happen when a model overfits data?
2. How can this be remedied?

Answer 23

1. Standard estimates of regression coefficients become inflated or unstable
2. Estimates can be stabilised (regularised) by adding a penalty to the estimating equations

For linear regressions, the penalty is added to the residual sum of squared errors (RSS)

𝑅𝑆𝑆(𝜆)= ∑(𝑦𝑖−𝑦̂𝑖 )2 +𝜆𝑓(𝛽)

In OLS we try to minimize RSS to find best optimal unbiased estimate of our regression coefficient

In penalized regression we add a penalty term called lambda x function of regression coefficient

Larger lambda the larger the penalty in our residual sums of squared error

Question 24

In principle with the right choice of lambda what can we get?

Answer 24

An estimator with a better MSE

Estimate is not unbiased but what we pay for in bias we make up for in variance

By sacrificing unbiasedness, we can reduce the variance to make the overall MSE lower

Question 25

We try to find a lambda(penalty) that does what?

Answer 25

Minimises error of unseen cases

If lambda is 0 then we have an OLS method and we have a bias of 0 if increase lambda then bias becomes much larger and MSE becomes larger but on other hand if our ordinary squared method or variance might be large but if increase lambda is becomes smaller

Question 26

What are 3 commonly used functions?

Answer 26

1. Ridge penalty

𝑓(𝛽)= ∑𝛽^2

sum of squared coefficients (∑𝛽2) forms the penalty
Also called L2 norm as squared

2. LASSO (Least Absolute Shrinkage and Selection Operator):
   𝑓(𝛽)= ∑|𝛽|

sum of absolute coefficients (||) forms the penalty

Also called L1 norm as its beta to the power of one

3. Elastic net
   – a combination of L1 and L2 norm r egularization

Question 27

What is commonly used to deal with ill conditioned regression problems such as multi-collinearity (high correlation between predictor variables) and
number of variables (p) is close to the sample size (n)?

Answer 27

Ridge regression

Question 28

How is ridge regression of β obtained?

Answer 28

By minimising root sum of squares plus our penalty which is lambda times sum of squared regression coefficients, this is penalty term which we add to our root RSS and we now call this RSS

The parameter λ scales the norm - controls the amount of penalty

Question 29

What is one of the important problems in applying ridge regression?

Answer 29

To choose the right value of λ

Question 30

What does LASSO (Least Absolute Shrinkage and Selection Operator) a promising technique for?

Answer 30

Variable selection

Finding a small subset of most predictive variables in a high dimensional dataset is an interesting and important problem

Question 31

How does LASSO tend to deal with overfitting?

Answer 31

Tends to assign zero coefficients to most irrelevant or redundant variables - This is also called a sparse solution

Question 32

How are LASSO estimates obtained?

Answer 32

Minimised RSS plus lasso penalty term which is lambda times absolute value of regression parameters

This is called L1 penalty/norm

Lasso penalty involves absolute values of regression parameters and not sum of the squared values like in ridge regression

Need to find best lambda to minimise penalised root mean squared error

Similar to ridge regression, the penalty parameter (λ) controls the amount of penalty (user customisable)

Question 33

If compute lasso or ridge data must be…

Answer 33

Standardised so those with large range do not dominate model selection:

Different units (m versus km) would result in different solutions

This is automatically done in most software packages

R packages such as “Glmnet” back transforms final regression coefficient on original scale!

Question 34

What is the z-transformation formula?

Answer 34

Linear transformation of values to common mean of zero and stand deviation of 1:
zi = (𝑥𝑖−𝑥̅)/𝑠

with
𝑧𝑖=𝑧 𝑡𝑟𝑎𝑛𝑠𝑓𝑜𝑟𝑚e𝑑 𝑜𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛
𝑜𝑓 𝑐𝑎𝑠𝑒 𝑖 𝑜𝑓 𝑡ℎ𝑒 𝑠𝑎𝑚𝑝𝑙𝑒

𝑥̅=𝑠𝑎𝑚𝑝𝑙𝑒 𝑚𝑒𝑎𝑛
𝑥𝑖=𝑜𝑟𝑔𝑖𝑛𝑎𝑙 𝑣𝑎𝑙𝑢𝑒 𝑜𝑓 𝑐𝑎𝑠𝑒 𝑖

𝑠=𝑠𝑡𝑎𝑛𝑑𝑎𝑟𝑑 𝑑𝑒𝑣𝑖𝑎𝑡𝑖𝑜𝑛 𝑜𝑓 𝑠𝑎𝑚𝑝𝑙𝑒

Question 35

The z-transformation changes the form of the distribution, it only adjusts the mean and the standard deviation!

True or false

Answer 35

FALSE

z-transformation does not change the form of the distribution, it only adjusts the mean and the standard deviation!

Question 36

How do we select lambda?

Answer 36

Goal is to evaluate the model in terms of it’s ability to predict future observations:

The model need to be evaluated on a dataset that was not used to build the model (test sets)

We assess different lambdas and choose the one which predicts best unseen cases using cross-validation

This best lambda is used fit the model using the complete data set

Take average MSE, and calculate mse of 100 different lambdas of different strength and pick lamda with smallest MSE

We pick the lambda which best predicts unseen cases (= smallest Mean squared error, MSE)

Question 37

What is the performance of our model function measured by?

Answer 37

A loss function for penalizing error in prediction.

Question 38

What is a loss of function measured by?

Answer 38

How good a prediction model does in terms of being able to predict the expected outcome..

Question 39

What is a popular loss of function?

Answer 39

MSE loss function

We decide to choose the function f(x) which minimizes the expected loss or here the expected mean squared prediction error (MSE).

The expected MSE can be estimated by cross-validation or bootstrapping methods - We use the same methodology as for internal validation!

Question 40

If build optimal predictive models any sensible subset selection algorithm can be combined with what?

Answer 40

Cross-validation to build a good prediction model

The idea is to build a large number of alternative models (of varying complexities) and evaluate the predictive performance using cross-validation to select the best model

In regularized regression we compare models with different lambdas!

Question 41

Using hold-out data for prediction accuracy estimation involves what?

Answer 41

Using CV to select optimal 𝜆 selects the best set of predictors of unseen cases.

However: Prediction accuracy measures are over-optimistic estimates for accuracy of future sample: CV test data were used to select our model!

Question 42

What is ridge not useful for?

Answer 42

Parsimonious model selection

Question 43

Ridge penalty function is very flat near the zero values of β, what does this mean?

Answer 43

Does not encourage the β coefficients to be exactly zero
Not good for variable selection
Not good for sparse problems

Alternative penalised methods (e.g., LASSO, see next) is a better option for variable selection

Question 44

What is lambda.1se?

Answer 44

This is a slightly stronger penalty than the minimum lambda and lies within one standard error of the optimal value of lambda.

The purpose of regularization is often to balance accuracy and simplicity: We want a model with the smallest number of predictors that also gives a good accuracy.

Setting lambda = lambda.1se results in a simpler model compared to lambda.min (less variables are selected), but the model might be a little bit less accurate than the one obtained with min.lambda.

Research suggest that this lambda sometimes predicts better in external data sets and selects less false positive predictors.

Question 45

When should you compute OLS?

Answer 45

If you have large sample sizes with a relative small number of variables of likely predictors (theory-driven)

Question 46

When should you compute Ridge?

Answer 46

If you expect many small effect sizes and predictors are likely true ones (you want to keep all variables in the model).

Question 47

When should you compute Lasso?

Answer 47

If you have a few stronger predictors among a large number of likely weak predictors or noise variables.

Question 48

What is not very meaningful in penalised regressions and why?

Answer 48

Statistical Inference of regression coefficients

This is because the penalised estimates are biased towards zero

Standard Error (SE) of penalised coefficients give only partial information of the precision

SE ignores the inaccuracy caused by bias

Software packages do not supply standard errors (SE), confidence interval (CI), or p-values for penalised regression. Internal validation is our “test”

Major aim in penalised regression is to build a prediction model/variable selection rather than performing statistical inference

Question 49

Regularized or penalized regressions are extensions of the linear model.

True or false

Answer 49

True

Question 50

Regularized or penalized regressions seek to do what?

Answer 50

Minimise the sum of the squared error (or MSE) of the model on the training data but also to try avoiding over-fitting by reducing the complexity of the model at the cost of some bias

This is done by shrinking the regression coefficients

Question 51

What are two popular examples of Regularized or penalized regressions?

Answer 51

Ridge Regression: where Ordinary Least Squares is modified to also minimize the squared absolute sum of the coefficients (called L2 regularization).

Lasso Regression: where OLS is modified to also minimize the absolute sum of the coefficients (L1 regularization).
Unlike Ridge, Lasso regression performs variable selection by shrinking some coefficients to 0