Multiple Regression - Model Selection

Question 1

How does a simultaneous regression model differ from hierarchical regression?

Answer 1

Simultaneous regression - all predictors are added at the SAME time (in a single step)

This means that the 1st variable we enter soaks up all of the variability – so if we entered physical health first, it soaked up the variability, leaving the left-overs for mental health. When we remove physical health, mental health all of a sudden is significant.

In hierarchical regression, predictors that are most important are entered first, then step 2 will include the first and 2nd; etc.

Question 2

What affects the attribution of shared variability in simultaneous and hierarchical regression?

Answer 2

The way the variables are entered (in their order).

Question 3

Why would we use the hierarchical model?

Answer 3

If we want to see the effects over and above a covariate - we will enter the uninteresting predictors first.

ex)If we add SES (covariate) in the model first, we can see what we’re actually interested in, after controlling for SES.

Question 4

If there’s no theory about predictors, what do we do?

Answer 4

We run all regressions!

We first evaluate all possible regressions, or employ a selection algorithm.

Question 5

If we compare all possible regressions (2 to the k) involving a set of predictors in a search for the “best” model, on what criteria might we compare the models? (5)

Answer 5

1. Compare models to R² (largest)- problem is Inflation (model with more predictors always looks better)
2. Adj R2 (largest)= We are penalized by adding predictors that aren’t worth their weight in df.
3. Press (Smallest)- Prediction sum of squares deleting 1 residual, computing new sum of squares…
4. Mallow CP (smallest)- Addresses the error of the fitted values and predictor after removal of residuals. The expected value of Mallow CP is 1 + the predicted value we have.
5. Parsimony - Less predictors the better

Question 6

What does a significant R2∆ change indicate in a hierarchical regression output?

ex)
Step 1 R2∆ is significant.

Step 2 R2∆ is not significant.

Step 3 R2∆ is significant.

Answer 6

Step 1, or the physical health contributes to the model significantly

Step 2, or physical and mental health does not contribute to the model significantly

Step 3, or physical, mental health and stress contributes to the model significantly

Question 7

In regards to degrees of freedom 2 (df2) in the hierarchical r output, how does it go up and down?

Answer 7

As we add more predictors, the denominator DF (n-k-1) goes down.

Question 8

As we add more predictors to a model, what happens to the R2?

Answer 8

The R2 value inflates- making it look like we have more effect when we probably don’t.

Question 9

What does selection algorithms do?

Answer 9

We can compare 2 predictors each time, instead of using simultaneous or hierarchical regression… this way, we don’t inflate the R2 value and we might actually see where the differences are without any inflation.

Question 10

What’s the formula for Adjusted R2?

Answer 10

1 - (n-1)(1-R2) ÷ n-k-1

or by doing

SSres ÷ SStotal x (dfTotal ÷ dfRes)

Question 11

What is the major disadvantage to the all possible regression approach?

Answer 11

It is time-consuming. Evaluating 32 models by just having 5 predictors (4,096 times!)… is daunting. The difference between R2 and adjusted R2 would be very large.

Question 12

What is the function of the Model Selection Algorithm?

What does R use to compute selection algorithms?

What are the 3 types of getting the AIC?

Answer 12

It doesn’t compute all possible models-uses just an algorithm to develop a single “best” model.

R uses AIC

3 Types:

• Forward selection
• Backward elimination
• Stepwise

Question 13

What is the algorithm that R uses to compute the model selection algorithm?

Answer 13

It uses the AIC, or the Akaike Information Criterion.

Question 14

Why is AIC referred to as a measure of relative goodness of fit of a model?

Answer 14

AIC is a measure of model fit that can only be compared to the RELATIVE goodness of fit of a model - it is not a free-standing measure of model fit. Only when it’s compared to another model.

Question 15

What’s the difference between R² and AIC?

Answer 15

R² is an absolute model fit value, while AIC can only be used to compare it to other models with the same outcome variable.

Question 16

What’s the benefit of using AIC?

Answer 16

It’s a trade-off between model complexity and accuracy (2k, or 2 times the model predictors).

It penalizes us for adding predictors that don’t improve our model.

Question 17

Which is the preferred model in regards to AIC?

Answer 17

Preferred model has the smallest AIC - it’s the most accurate because the larger value of AIC indicates worse fit, corrected for the number of variables.

Question 18

How do the 3 types of model selection algorithms differ?

Answer 18

They differ in the way the variables are added and removed to the model.

Question 19

Describe forward selection.

Answer 19

Forward selection – start with the smallest model (usually Empty or Covariate), then r determines which model adding one predictor at a time produces the model with the smallest AIC.

Question 20

Describe backwards elimination.

Answer 20

Starts with full model that includes all predictors. Shows which predictor can be removed to produce the smallest AIC.

Question 21

What’s the problems with the forward selection and backward elimination algorithms?

Answer 21

Due to shared variability, a previously added or removed predictor could become a bad good or bad predictor.

The problem is that good or bad predictors cannot be added or removed from these models because for forward selection, we can only add, and backward elimination, we can only remove.

Question 22

Describe Stepwise selection.

Answer 22

Stepwise combats the limitations of forward and backwards by removing and adding predictors that will result in the model with the lowest AIC.

Question 23

How do we address the model selection algorithms and its capitalization on chance?

What do I mean by capitalization on chance?

Answer 23

All of the model selection algorithms (AIC) are all sample-based estimates of R², therefore it overestimates population R². Adjusted R² addresses the capitalization on chance by indicating the loss of predictive power or shrinkage.

Shrinkage:
We can address this issue by applying the prediction equation developed in 1 sample to a second sample which will result in a smaller R² (calculated as the squared correlation bw Y and y-hat), providing a better estimate of the population.

Question 24

How can we adjust for Shrinkage?

Answer 24

• Cross-validate- collect data from 2nd sample to see if it matches
• Double cross-validate: apply estimate model from old sample to new and new to old.
• Data Splitting – split a large dataset in half and cross-validate within the dataset