3. Testing and Evaluating Linear Models

Question 1

What are the three parts of evaluation?

Answer 1

Evaluating individual coefficients, evaluating overall model quality, evaluating model assumptions

Question 2

What question would ask to explore significant of individual effects?

Answer 2

Is our model predictor informative of the relationship between x and y?

Question 3

How do we evaluate individual coefficients?

Answer 3

Hypothesis is needed to make the data testable

Question 4

What are the steps involved in hypothesis testing?

Answer 4

Research question

Statistical hypothesis

Calculate estimate of effect of interest

Calculate appropriate t-statistic

Evaluate t-statistic against the null

Question 5

What should a good research question include?

Answer 5

Constructs under study
the relationship being tested
A direction of relationship
Target populations etc.

Question 6

What are the different types of hypothesis?

Answer 6

Null = Precise and states specific value for the effect of interest

Alternative = Not specific, states something other than null is more likely to occur

H0 = B1 = 0
H1 = B1 not = 0

Question 7

What would a null hypothesis suggest about the relationship between x and y?

Answer 7

If x and y are unrelated, change in x will not result in any change in y do b1 will be equal to 0

Question 8

What is a p-value?

Answer 8

The P value means the probability, for a given statistical model that, when the null hypothesis is true.

E.g. P < 0.05 is the probability that the null hypothesis is true so in this case we would reject the null

Question 9

What is a t-statistic?

Answer 9

T is simply the calculated difference represented in units of standard error.

A test statistic describes how closely the distribution of your data matches the distribution predicted under the null hypothesis of the statistical test you are using so if it is a larger the number, it is further away from what the null hypothesis would predict it to be.

Predicted value of beta/SE of predicted beta

(the smaller the SE, the more precise)

The greater the magnitude of T, the greater the evidence against the null hypothesis.

Question 10

How do we actually test the statistical significance of individual coefficients?

Answer 10

We select a significance level, α (typically .05)

Then we calculate the p-value associated with our test statistic (here β)

If the associated p is smaller, then we reject the null.

If it is larger, then we fail to reject the null.

Question 11

What does it mean if the p-value is < t-stat?

Answer 11

Reject the null

Question 12

What does it mean if the p-value is > t-stat?

Answer 12

Fail to reject the null

Question 13

What sampling distribution is used for the null hypothesis?

Answer 13

T-distribution - n-k-1 degrees of freedom

Need significance level and critical value to compare observed t-value

Question 14

What is a critical value?

Answer 14

Establishes regions in sampling distribution of test statistic = Used to calculate upper and lower bounds of CI

Question 15

What are the different factors that impact SE value?

Answer 15

SE is smaller when residual variance (SS Residual) is smaller
SE is smaller when sample size ( N ) is larger
SE is larger when the number of predictors (k) is larger
SE is larger when a predictor is strongly correlated with other predictors ( R2xj)

Question 16

What is a t-distribution?

Answer 16

Standardized differences to sample means to population mean when population SD isn’t known

Normally distributed population

Question 17

What is the confidence level for null?

Answer 17

1 - alpha

Question 18

What does it mean if the confidence interval doesn’t include 0?

Answer 18

If it doesn’t include 0, then it is significant

Question 19

How can we compare the critical value and t-statistic to tell us if we can reject the null or not?

Answer 19

If the value of the test statistic is less extreme than the critical value, then the null hypothesis cannot be rejected.

Absolute value of t-statistic > critical value = Reject the null

Question 20

When are we more likely to find a significant effect?

Answer 20

When we have picked good variables (smaller residual SS) and we have a large sample

Question 21

How do we evaluate overall model performance?

Answer 21

The aim of our linear model is to build a model which describes y as a function of x.

That is we are trying to explain variation in y using x so we evaluate model evaluation via assessing variation.

Question 22

What does variation in y stand for?

Answer 22

Total variation of interest

Question 23

What is variation made up of?

Answer 23

Model and Residual Variance

Question 24

How do we measure total variation in the outcome?

Answer 24

Sum of Squares = SS model + SS residual

Question 25

What does R2 mean (coefficient of determination)?

Answer 25

Quantifies the amount of variability in the outcome accounted for by the predictors.

More variance accounted for, the better.

Represents the extent to which the prediction of y is improved when predictions are based on the linear relation between x and y.

R2 = SSmodel/SStotal or 1 - (SSresidual/SStotal)

Question 26

What is adjusted R2?

Answer 26

It is the R2 value adjusted for when there are two or more predictors

Random sampling can impact it

Adjusted for sample size and number of predictors

Increased IVs = ^ Value

Question 27

Why is it important to compute adjusted R2 in a model with multiple predictors?

Answer 27

It accounts for random fluctuation that comes with increases in sample size & number of predictors

Question 28

What does comparing R2 and adjusted R2 tell us?

Answer 28

The most vital difference between adjusted R-squared and R-squared is simply that adjusted R-squared considers and tests different independent variables against the model and R-squared does not.

So if big difference between them - i.e adjusted R2 is a lot smaller then the additional input variables are not adding value to the model.

Question 29

If there is a smaller sample, what does this mean for fluctuations in adjusted R2?

Answer 29

In smaller samples , the fluctuations from zero will be larger on average.

Question 30

If we have a highly correlated predictor, how does that impact the SE of coefficients?

Answer 30

Increases SE as we’re less certain that our variables are driving the effect