Linear Model Evaluation/Diagnostics

Question 1

What is the R^2 statistic?

Answer 1

The proportion of variance explained by the model

Question 2

Why is R^2 not always the best indicator of predictive power?

Answer 2

An overfitted model will have great R^2 statistic but poor predictive power
High variance, then low R^2 score even if correct model

Question 3

What are the assumptions we make about the errors in a model?

Answer 3

• well described by a normal distribution
• have constant variance
• are independent of each other

Question 4

What do we expect to see when remove the signal from the model?

Answer 4

Residuals that are normally distributed

Question 5

What are the two qualitative ways to assess normality?

Answer 5

• look at histogram of the residuals

- a QQ Norm plot of the residuals

Question 6

What are the two quantitative ways to assess normality?

Answer 6

• Wilk Shapiro test for Normality

- Kolmogorov Smirnov test for Normality

Question 7

Describe QQ Norm plots

Answer 7

• Plot the quantiles of two sets of data against each other
• If there shapes are similar and roughly normally distributed, tend to get a straight line
• plots the residuals sorted in order, against the standardised quantiles for the distribution of interest

Question 8

What is the ith point of QQ Norm plots typically given by?

Answer 8

i/(n+1)

Question 9

Describe the Shapiro Wilks Test

Answer 9

Produces a statistic which relates to the straightness of the QQ plot
Null hypothesis, H0: data are normally distributed

Question 10

What will happen if the assumption that the errors are independent is violated?

Answer 10

Standard errors and p values are systematically too small and risk drawing the wrong conclusions about model covariates

Question 11

What can the null hypothesis of uncorrelated errors be formally tested by?

Answer 11

Durbin Watson test

Question 12

What can independence also be violated by?

Answer 12

Philosophical ways, like pseudoreplication

Question 13

What is the practical consequence of falsely assuming independence?

Answer 13

Can conclude that one or more unrelated variables are genuinely related to the response

Question 14

What can we do if we have correlation in the residuals?

Answer 14

• Ignore the correlation in residuals
• Try to remove the correlation in model residuals by sub-setting the data
• Account for the correlation using, a generalized least squares model

Question 15

What do we use partial residual plots for?

Answer 15

To address if non-linearity of predictors is caused by either predictor or another unknown one

Question 16

What do partial residual plots show?

Answer 16

Residuals and relationships between y and individual x with adjustments for other x’s

Question 17

What are the useful diagnostic properties of partial residuals?

Answer 17

• The slope of the line is the regression coefficent
• The extent of the scatter tells us about the support for the function
• We can identify large residuals
• Curved plots signal non-linear relationships

Question 18

What do we do if we have error distribution shape problems?

Answer 18

• Try transforming things to address the distributional shape problems
• Move to other models like generalised linear models
• Bootstrap your way to glory

Question 19

What do we do if we have independence problems?

Answer 19

• Move to other models/methods like mixed models (LMM, GLMM) or used generalised estimating equations

Question 20

What do we do if we have signal problems?

Answer 20

• Use complex linear models or generalised additive models (GAM)

Question 21

How do we bootstrap?

Answer 21

1. We make a new dataset of the same dimension, by sampling the rows of the data with replacement
2. We do this a lot and fit models at each stage
3. This shows how roughly things might change if we were to have another sample of data