Term 2: Lecture 9 Multiple Linear Regression Flashcards
In simple linear regression, we predict a dependent variable Y using a single independent variable X.
The statistical model is expressed by the equation:
where:
• b0
- b1
- ei
From this model we can derive a prediction of
Y= bo + b1Xi + ei
is the intercept of the regression line – also called the constant;
is the slope of the regression line - also called the regression coefficient;
is the residual (the prediction error specific to each individual).
Y given X:
Y(hat) = bo + b1Xi
We may extend the simple linear regression model by using, what?
more than one independent variable (lots of Xs)
where X1, X2, and X3 are independent variables (______) that vary by _________i .
predictors
individual
(this means the different individuals
have different values on these independent variables)
Compared to last week, we have added one dimension, and our equation now describes not a line, but…….
a plane (a flat surface) that lies in the three-dimensional space.
the Coefficients table displays….
(USC, SC, T, sig)
The unstandardized coefficient for RAC (0.433) represents…..
The t-test for each predictor tests………
The standardized coefficients are a measure of the….
Coefficients
needed for the estimated regression
equation: 𝐵𝑁𝑃 (hat i) = −2.197 + 0.433 × 𝑅𝐴𝐶𝑖 + 0.410 × 𝑊𝐷𝑖
how much BNP is predicted to rise if RAC increases by 1, holding WD constant (or, more generally, holding all other independent variables constant).
The analogous statement holds for the coefficient of WD.
THE T- TEST
the hypothesis that the unstandardized coefficient is zero in the population.
Here, the two coefficients (one for each of the two independent variables) are significantly different from zero. So there is evidence that Racism and White Defence are both predictive of Affinity for the BNP (in the presence of each other)
Standardized Coeffients:
relative importance of the predictors. They represent the standardized contribution of the X-variables to the prediction of Y – that is, irrespective of the measurement scale of X.
.
how do you assess the model fit of a multiple regression
Model Summary table in SPSS
R:
R2:
Standard Error of the Estimate [SEE]:
Another way to think of the SEE is….. SD R
The multiple correlation coefficient: the multiple correlation of Y with all independent variables.
The Coefficient of Determination: the proportion of variance in Y that is explained by the predictors.
This is (approximately) the average error of prediction.
If we use the regression equation to predict
the value of Y, then our average error will be the value of SEE. The SEE is an indicator of the precision of our prediction.
that it is approximately the standard deviation of the residuals.
Adjusted R2: dcoR
it adjusts for….
Adding a predictor to a model will never….. but may ______ _________ even if the predictor has no predictive power
This is a downward correction of R2.
for the number of predictors in a model and is useful for comparing different models (with different numbers of predictors).
reduce R2, but may increase it even if the predictor has no predictive power.
Adjusted R2 penalises a model for having predictors that do not contribute much to model fit.
The larger the number of predictors, the smaller Adj. R2 gets, relative to R2.
To understand how Adj. R2 is computed, we need to consider the
ANOVA table…
In the context of regression, the ANOVA table reports….
The ANOVA table here divides the total sum of squares of Y into two components:
• Regression Sum of Squares:
• Residual Sum of Squares:
…..whether the model, as a whole, makes a statistically significant contribution to improving the prediction of the dependent variable.
The variation of Y that is “accounted
for” by the model.
The variation of Y that the model does
not account for.
The method of linear regression assumes that relationships between variables can be represented by straight lines. This does not seem a plausible assumption for categorical variables.
How can we enter Age into the regression equation?
what type of data is it?
what are they called?
add a dummy variable
We can enter categorical variables into the regression equation when they have only two categories, which are coded “0” and “1”.
Variables of this kind are called “dummy variables”. We can recode the variable “Age groups” into two separate dummy variables, according to the following coding scheme:
We can enter categorical variables into the regression equation when they have only two categories, which are coded “0” and “1”. Variables of this kind are called “dummy variables”. We can recode the variable “Age groups” into two separate dummy variables,
according what following coding scheme:
We enter the two dummy variables into the regression equation, as…….
Respondents under 30 are represented by Dummy 1.
• Respondents between 30 and 49 are represented by
Dummy 2.
• Respondents 50 or older form the reference category.
as predictors.
It is of interest to establish whether adding predictors (here: the two age-dummies) to the model improves model fit (i.e., improves how well
our predicted values represent the observed values of the dependent
variable).
Using SPSS, we can test the null hypothesis that Model 1 (with two
predictors, RAC and WD) has the same model fit as Model 2 (with four
predictors: RAC, WD, YOUNG and MIDAGED) in the population.
Adding the two variables “Age: under 30” and “Age: 30-49” to the model
results in an R2-change of .071.
The F-test tests the null hypothesis that Model 2 is no better than Model 1
at predicting the dependent variable. The associated test statistic is F =
8.477, with degrees of freedom 2 and 156. Since p<0.001 (p=0.000 to
three decimal points), the result is statistically significant, and we reject
the null hypothesis.
So we have evidence that adding the two variables “Age: under 30” and
“Age: 30-49” improves our prediction of “Affinity for BNP”. We can
conclude that age is a predictor of “Affinity for BNP” in the population.
Principles of Model Comparison
A good statistical model should:
• fit the data well (result in a good prediction of the dependent
variable), and
• be parsimonious (not contain more independent variables
than necessary).
In practice, there is often a trade-off between goodness-of-fit
and parsimony. It is often possible to improve the fit slightly by
adding additional independent variables, even if these additional
variables have only very small predictive power. This increases
the danger of overfitting - making the model too complex and
tailored too closely to the specific dataset on which it was
created. The resulting model is usually not reproducible in new
samples.
Model Comparison in Practice
Adding one or more independent variables to a model will usually ______ R2 and (it can happen that r2 stays the same)
this mean that R2 is ____ ______ for the model selection
Loo
increase R2, and adding independent variables will never reduce R2 (it can
happen that R2 stays the same).
This means that R2 is not useful for model selection. Looking at R2, the
‘bigger’ model will always seem ‘better’. Put differently, R2 only
considers goodness of fit, but does not consider parsimony.
Adjusted R2 is more useful for model comparison, since it includes a
‘penalty’ (downward adjustment) for the number of independent
variables. Adjusted R2 thus takes into account both goodness of fit and
parsimony.
In our example,
Adj. R2 for Model 1 is .263
Adj. R2 for Model 2 is .326
So, adding the ‘age dummies’ has resulted in an increase in model fit
that outweighs the loss of parsimony.
how to write up multiple regression
The results show that “Racism”, “White Defence”, and “Age” are
predictors of attraction to the BNP. People with racist attitudes, people
who feel that they are disadvantaged as ‘whites’, and younger people
are more likely to be attracted to the BNP than non-racists and older
people. Overall, the model accounts for about a third (34%) of the
variance in “Affinity for the BNP”.
