Week 4: Multiple Regression Flashcards
What is the decision tree for multiple regression? - (4)
- Continous
- Two or more predictors that are continous
- Multiple regression
- Meets assumptions of parametric tests
simple linear regression
the outcome variable Y is
predicted using the equation of a straight line
Multiple regression still uses the same basic equation of …. but the model is still complex
Multiple regression is the same as simple linear regression expect for - (2)
every extra predictor you include, you have to add a coefficient;
so, each predictor variable has its own coefficient, and the outcome variable is predicted from a combination of all the variables multiplied by their respective coefficients plus a residual term
Multiple regression equation
In multiple regression equation, list all the terms - (5)
- Y is the outcome variable,
- b1 is the coefficient of the first predictor (X1),
- b2 is the coefficient of the second predictor (X2),
- bn is the coefficient of the nth predictor (Xn),
- εi is the difference between the predicted and the observed value of Y for the ith participant.
Multiple regression uses the same principle as linear regression in a way that
we seek to find the linear combination of predictors that correlate maximally with the outcome variable.
Regression is a way of predicting things that you have not measured by predicting
an outcome variable from one or more predictor variables
Regression can be used to produce a
linear model of the relationship between 2 variables
Record company interested in creating model of predicting recording sales from advertising budget and plays on radio per week (airplay)
- Example of it’s MR plotted on + number of vars measured, what vertical axis shows, horizontal and third axis shows - (4)
It is a three dimensional scatter plots, which means there are three axes measuring the value of the three variables.
The vertical axis measures the outcome, which in this case is the number of album sales.
The horizontal axis measures how often the album is played on the radio per week.
The third axis, which can can think of being directed into the page measures the advertising budget.
Can’t plot a 3D plot of MR as shown here
for more than 2 predictor (X) variables
The overlap in the diagram is the shared variance, which we call the
covariance
covariance is also referred to as the variance
shared between the predictor and outcome variable.
What is shown in E?
The variance in Album Sales not shared by the predictors
What is shown in D?
Unique variance shared between Ad Budget and Plays
What is shown in C?
The variance in Album Sales shared by Ad Budget and Plays
What is shown in B?
Unique variance shared between Plays and Album Sales
What is shown in A?
Unique variance shared between Ad Budget and Album Sales
If you got two prediictors thart overlap and correlate a lot then it is a .. model
bad model can’t uniquely explain the outcome
In Hierarchical regression, we are seeing whether
one model explains significantly more variance than the other
In hierarchical regression predictors are selected based on
past work and the experimenter
decides in which order to enter the predictors into the model
As a general rule for hierarchical regression, - (3)
known predictors (from other research) should be entered into the model first in order of their importance in predicting the outcome.
After known predictors have been entered, the
experimenter can add any new predictors into the model.
New predictors can be entered either all in one go, in a stepwise manner, or hierarchically (such that the new predictor
suspected to be the most important is entered first).
Example of hierarchical regression in terms of album sales - (2)
The first model allows all the shared variance between Ad budget and Album sales to be accounted for.
The second model then only has the option to explain more variance by the unique contribution from the added predictor Plays on the radio.
What is forced entry MR?
method in which all predictors are forced
into the model simultaneously.
Like HR, forced entry MR relies on
good theoretical reasons for including the chosen predictors,
Different from HR, forced entry MR
makes no decision about the order in which variables are entered.
Some researchers believe that about forced entry MR that
this method is the only appropriate method for theory testing because stepwise techniques are influenced by random variation in the data and so rarely give replicable results if the model is retested.
How to do forced entry MR in SPSS? - (4)
Analyse –> Linear –> Regression
Put outcome in DV and IVs (predictors, x) in IV box
Can select a range of statistics in statistics box and press okay to check colinearity assumption
Can also click plots to check assumptions of homoscedasticity and lineartiy
Why select colinearity diagnostics in statistics box for multiple regression? - (2)
This option is for obtaining collinearity statistics such as the
VIF, tolerance,
Checking assumption of no multicolinearity
Multicollinearity exists when there is a
strong correlation between two or more predictors in a regression model.
Multicollinearity poses a problem only for multiple regression because
simple regression requires only one predictor.
Perfect collinearity exists in multiple regression when at least
e.g., two predictors are perfectly correlated , have a correlation coefficient of 1
If there is perfect collinearity in multiple regression between predictors it
becomes impossible
to obtain unique estimates of the regression coefficients because there are an infinite number of combinations of coefficients that would work equally well.
Good news is perfect colinearity in multiple regression is rare in
real-life data
If two predictors are perfectly correlated in multiple regression then the values of b for each variable are
interchangable
The bad news is that less than perfect collinearity is virtually
unavoidable
As colinearity increases in multiple regression, there are 3 problems that arise - (3)
- Untrustory bs
- Limit size of R
- Importance of predictors
As colinearity increases, there are 3 problems that arise - (3)
importance of predictors - (3)
Multicollinearity between predictors makes it difficult
to assess the individual importance of a predictor.
If the predictors are highly correlated, and each accounts for similar variance in the outcome, then how can we know
which of the two variables is important?
Quite simply we can’t tell which variable is important – the model could include either one, interchangeably.
One way of identifying multicollinearity in multiple regression is to scan a
a correlation matrix of all of the predictor
variables and see if any correlate very highly (by very highly I mean correlations of above .80
or .90)
SPSS produces colinearity diagnoistics in multiple regression which is - (2)
variance inflation factor (VIF) and tolerance
The VIF indicates in multiple regression whether a
predictor has a strong linear relationship with the other predictor(s).
If VIF statistic is above 10 in multiple regression there is a good reason to worry about
potential problem of multicolinearity
If VIF statistic above 10 or approaching 10 in multiple regression then what you would want to do is have a - (2)
look at your variables to see if you need to include all variables whether all need to go in model
if high correlation between 2 predictors (measuring same thing) then decide whether its important to include both vars or take one out and simplify regression model
Related to the VIF in multiple regression is the tolerance
statistic, which is its
reciporal (1/VIF) = inverse of VIF
In tolerance, value below 0.2 shows in multiple regression
issue with multicolinerity
In Plots in SPSS, you put in multiple regression - (2)
ZRESID on Y and ZPRED on X
Plot of residuals against predicted to asses homoscedasticity
What is ZPRED in MR? - (2)
(the standardized predicted values of the dependent variable based on the model).
These values are standardized forms of the values predicted by the model.
What is ZRESID in MR? - (2)
(the standardized residuals, or errors).
These values are the standardized differences between the observed data and the values that the model predicts).
SPSS in multiple linear regression gives descriptive outcoems which is - (2)
- basics means and also a table of correlations between variables.
- This is a first opportunity to determine whether there is high correlation between predictors, otherwise known as multi-collinearity
In model summary of SPSS, it captures how the model or models explain in MR
variance in terms of R squared, and more importantly how R squared changes between models and whether those changes are significant.
Diagram of model summary
What is the measure of R^2 in multiple regression
measure of how much of the variability in the outcome is accounted for
by the predictors
The adjusted R^2 gives us an estimate of in multiple regression
fit in the general population
The Durbin-Watson statistic if specificed in multiple regresion tells us whether the - (2)
assumption of independent errors is tenable (value less than 1 or greater than 3 raise alarm bells)
value closer to 2 the better = assumption met
SPSS output for MR = ANOVA table which performs
F-tests for each model
SPSS output for MR contains ANOVA that tests whether the model is
significantly beter at predicting the outcome than using the mean as a ‘best guess’
The F-ratio represents the ratio of
improvement in prediction that results from fitting the model, relative to the inaccuracy that still exists in the model
We are told the sum of squares for model (SSM) - MR regression line in output which represents
improvement in prediction resulting from fitting a regression line to the data rather than using the mean as an estimate of the outcome
We are told residual sum of squares (Residual line) in this MR output which represents
total difference between
the model and the observed data
DF for Sum of squares Model for MR regression line is equal to
number of predictors (e.g., 1 for first model, 3 for second)
DF for Sum of Squares Residual for MR is - (2)
Number of observations (N) minus number of coefficients in regression model
(e.g., M1 has 2 coefficents - one for predictor and one for constant, M2 has 4 - one for each 3 predictor and one for constant)
The average sum of squares in ANOVA table is calculated by
calculated for each term (SSM, SSR) by dividing the SS by the df. T
How is the F ratio calculated in this ANOVA table?
F-ratio is calculated by dividing the average improvement in prediction by the model (MSM) by the average
difference between the model and the observed data (MSR)
If the improvement due to fitting the regression model is much greater than the inaccuracy within the model then value of F will be
greater than 1 and SPSS calculates exact prob (p-value) of obtaining value of F by change
What happens if b values are positive in multiple regression?
there is a positive relationship between the predictor and the outcome,
What happens if the b value is negative in multiple regression?
represents a negative relationship between predictor and outcome variable?
What do the b values in this table tell us what relationships between predictor and outcome variable in multiple regression? (3)
Indicating positive relationships so as advertising budget increases, record sales increases (outcome)
plays on ratio increase as do record sales
attractiveness of band increases record sales
The b-values also tell us, in addition to direction of relationship (pos/neg) , to what degree each in multiple regression
predictor affects the outcome if the effects of all other predictors are held constant:
B-values tell us to what degree each predictor affects the outcome if the effects of all other predictors held constant in multiple regression
e.g., advertising budget - (3)
(b = 0.085):
This value indicates that as advertising budget (x)
increases by one unit, record sales (outcome, y) increase by 0.085 units.
This interpretation is true only if the
effects of attractiveness of the band and airplay are held constant.
Standardised versions of b-values are much more easier to interpret as in muliple regression
not dependent on the units of measurements of variables
The standardised beta values tell us that in multiple regression
the number of standard deviations that the outcome will change as a result of one standard deviation change
in the predictor.
The standardized beta values are all measured in standard deviation
units and so are directly comparable: therefore, they provide a in MR
a better insight into the
‘importance’ of a predictor in the mode
If two predictor variables (e.g., advertising budget and airplay) have virtually identical standardised beta values (0.512, and 0.511) it shows that in MR
both variables have a comparable degree of importance in the model
The confidence intervals of unstandardised beta values are boundaries constructed such that
95% of these sampels these boundaries containn true value of b
A good regression model will have a narrow and small CI interval indicating in MR
value of b in this sample is close to the true value of b in the populatio
A bad regression model have CI that cross zero indicating that in MR
in some samples the predictor has a negative
relationship to the outcome whereas in others it has a positive relationship
In image below, which are the two best predictors based on CIs and one that isn’t as (2) in MR
two best predictors (advertising and airplay) have very tight confidence intervals indicating that the estimates for the current model are likely to be representative of the true population values
interval for attractiveness is wider (but still does not cross zero) indicating that the parameter for this variable is less representative, but nevertheless significant.
If you do part and partial correlations in descriptive box, there will be another coefficients table which looks this in MR like:
The zero-order correlations are the simple in MR
Pearson’s correlation coefficients
The partial correlations represent the in MR
represent the relationships between each predictor and the outcome variable, controlling for the effects of the other two predictors.
The part correlations in MR - (2)
represent the relationship between each predictor and the outcome, controlling for the effect that the other two variables have on the outcome.
representing the unique relationship each predictor has with otucome
In this table , zero-order correlation is calculated by - (2)
variance of outcome explained by predictors divided by total
(A+C)/(A+B+C+E)
Partial correlations in example is calculated by in MR- (2)
unique variance in outcome (ignore all other predictors) explained by predictor divided by variance in outcome not explained by all other predictors
A/A+E
Part correlations are calculated by - (2) in MR
unique variance in outcome explained by predictor divided by total variance in outcome
A/A+B+C+E
How to interpret this image in terms of colinearity - VIF and tolerance in MR
For our current model the VIF values are all well below 10 and the tolerance statistics all well above 0.2;
therefore, we can safely conclude that there is no collinearity within our data.
We can produce casewise diagnostics to see a in MR to see (2)
summary of residuals statistics to be examined of extreme cases
To see whether individual scores (cases) influence the modelling of data too much
SPSS casewise diagnostics shows cases that have a standardised residuals that are in MR (2)
less than -2 or greater than 2
(We expect about 5% of our cases to do tha and 95% to have standardised residuals within about +/- 2.)
What does this casewise diagnostic show? - (2) MR
- 99% of cases should lie within ±2.5 so expect 1% of cases lie outside limits
- From cases listed, clear two cases (1%) lie outside of limits (case, 164 [investigate further has residual 3] and 179) - 1% which isconform to accurate model
If there are many more cases we likely have (more than 5% of sample size) in case wise then in MR
broken the assumptions of the regression
If cases are a large number of standard deviations from the mean, we may want to in casewise diagnostics in MR
investigate and potentially remove them because they are ‘outliers’
Diagram of assumption of homoscedasticity and linearity of ZRESID againsr ZPRED in MR
What does this partial plot show? - (2) in MR
the partial plot shows the strong positive relationship to album sales.
There are no obvious outliers and the cloud of dots is evenly spaced out around the line, indicating homoscedasticity.
What does this plot show in MR(2)
the plot again shows a positive relationship to album sales, but the dots show funnelling,
There are no obvious outliers on this plot, but the funnel-shaped cloud indicates a violation of the assumption of homoscedasticity.
P plot and histogram of normally distributed in MR
P plot for skewed distirbution histogram for MR
If R^2 is 0.374 (outcome var in productivity and 3 predictors) then it shows that in MR
37.4% of the variance in productivity scores was accounted for by 3 predictor variables
This table tells us in terms of standardised beta values that (outcome is productivity in MR)
holidays had standardized beta coefficient of 0.031 whereas cake had a much higher standardized beta coefficient of 0.499 which tells us that amount of cake given out much better predictor of productivity than the amount of holidays taken
For pay we have a beta coefficient of 0.323 which tells us that pay was also a pretty good predictor in the model of productivity but slightly less than cake
What does this table tells us in terms of signifiance? - (3) in MR
- P value for holidays is 0.891 which is not significant
- P value for cake is 0.032 is significant
- P value for pay is 0.012 is significant
What does this image show in terms of VIF? in MR
o All below 10 here showing we are unlikely to have a problem with multicollinearity so we can not worry about that for this data
In ANOVA it is comparing M2 with all its predictor variables with in MR
baseline not M1
To see if M2 is an improvement of M1 in HR we need to look at … in model summary in MR
change statistics
What does this change statistic show in terms of M2 and M1 in MR
M2 explains an extra 7.5% which is sig
Each of these beta values shown in table has an associated standard error indicating to what extent and used to determine (2)
values would vary across different samples,
and these standard errors are used to determine
whether or not the b-value differs significantly from zero
For this output interpret whether predicotrs are sig predictors of record scales and magnitude t statistic on impact of record sales in MR - (2)
For this model, the advertising budget (t(196) = 12.26, p < .001), the amount of radio play prior to release (t(196) = 12.12, p < .001) and attractiveness of the band (t(196) =4.55, p < .001) are all significant predictors of record sales.
From the magnitude of the t-statistics we can see that the advertising budget and radio play had a similar impact,
whereas the attractiveness of the band had less impact.
Diagram of outliers
All residuals should lie within ….. SDs for no outliers /normal amount of outliers in MR
-3 and 3 SD
Which variables (if any) are highly correlated in MR?
Weight, Activity, and the interaction between them are statistically significant
What does homoscedasticity and hetrodasticity mean in MR? - (2)
Homoscedasticity: similar variance of residuals (errors) across the variable continuum, e.g. equally accurate.
Heteroscedasticity: variance of residuals (errors) differs across the variable continuum, e.g. not equally accurate
Diagram of normal, skewed to left (pos) and skewed to right (neg) of p-plots in MR
Diagram of excluded variables table in SPSS - (3) in MR
- Tells that OCD interpretiotn of intrustrions would have not have a significant impact on model’s ability to predict social anxiety
Beta value of Interpretation of Intrusions is very small, indicating small influence on outcome variable
Beta is the degree of change in the outcome variable for every 1 unit of change in the predictor variable.
When checking assumption fo regression, what does this graph tell you in MR
Normality of residuals
A consumer researcher was interested in what factors influence people’s fear responses to horror films. She measured gender and how much a person is prone to believe in things that are not real (fantasy proneness). Fear responses were measured too. In this table, what does the value 847.685 represent in MR
The residual error in the prediction of fear scores when both gender and fantasy proneness are included as predictors in the model.
A psychologist was interested in whether the amount of news people watch predicts how depressed they are. In this table, what does the value 3.030 represent in MR
The improvement in the prediction of depression by fitting the model
When checking the assumption of the regression, the following graph shows (hint look at axis titles)
Regression assumptions that have been met
A consumer researcher was interested in what factors influence people’s fear responses to horror films. She measured gender (0 = female, 1 = male) and how much a person is prone to believe in things that are not real (fantasy proneness) on a scale from 0 to 4 (0 = not at all fantasy prone, 4 = very fantasy prone). Fear responses were measured on a scale from 0 (not at all scared) to 15 (the most scared I have ever felt).
Based on the information from model 2 in the table, what is the likely population value of the parameter describing the relationship between gender and fear in MR
Somewhere between −3.369 and −0.517
A consumer researcher was interested in what factors influence people’s fear responses to horror films. She measured gender (0 = female, 1 = male) and how much a person is prone to believe in things that are not real (fantasy proneness) on a scale from 0 to 4 (0 = not at all fantasy prone, 4 = very fantasy prone). Fear responses were measured on a scale from 0 (not at all scared) to 15 (the most scared I have ever felt).
How much variance (as a percentage) in fear is shared by gender and fantasy proneness in the population in MR
13.5%
Recent research has shown that lecturers are among the most stressed workers. A researcher wanted to know exactly what it was about being a lecturer that created this stress and subsequent burnout. She recruited 75 lecturers and administered several questionnaires that measured: Burnout (high score = burnt out), Perceived Control (high score = low perceived control), Coping Ability (high score = low ability to cope with stress), Stress from Teaching (high score = teaching creates a lot of stress for the person), Stress from Research (high score = research creates a lot of stress for the person), and Stress from Providing Pastoral Care (high score = providing pastoral care creates a lot of stress for the person). The outcome of interest was burnout, and Cooper’s (1988) model of stress indicates that perceived control and coping style are important predictors of this variable. The remaining predictors were measured to see the unique contribution of different aspects of a lecturer’s work to their burnout.
Which of the predictor variables does not predict burnout in MR
Stress from research
Using the information from model 3, how would you interpret the beta value for ‘stress from teaching’ in MR
As stress from teaching increases by one unit, burnout decreases by 0.36 of a unit.
How much variance in burnout does the final model explain for the sample in MR
80.3%
A psychologist was interested in predicting how depressed people are from the amount of news they watch. Based on the output, do you think the psychologist will end up with a model that can be generalized beyond the sample?
No, because the errors show heteroscedasticity.
Diagram of no outliers for one assumption of MR
Note that you expect 1% of cases to lie outside this area so in a large sample, if you have one or two, that could be ok
Output diagram what does output show in MR? - (2)
Difference between no predictors and model 1 (a).
Difference between model 1 (a) and model 2 (b).
Our model 2 is significantly better at predicting the value of the outcome variable than the null model and model 1 (F (2, 197) = 167.2, p<.001) and explains 66% of the variance in our data (R2=.66
What does this output show in terms of regression model in MR? - (3)
y = 0.09x1 + 3.59x2 + 41.12
For every £1,000 increase in advertising budget there is an increase of 87 record sales (B = 0.09, t = 11.99, p<.001).
For every number of plays on Radio 1 per week there is an increase of 3,589 record sales (B = 3.59, t = 12.51, p<.001).
Report R^2, F statistic and p-value to 2DP for overall model - (3)
o R squared = 0.09
o F statistic = 22.54
o P value = p < 0.001
Report beta and b values for video games, resitrctions and parental aggression to 2DP and p-value in MR
Which of the following assumptions of homosecdasticity and linearity is correct?
AThere is non-linearity in the data
BThere is heteroscedasticity in the data
CThere is both heteroscedasticity and non-linearity in the data
DThere are no problems with either heteroscedasticity or non-linearity
D - data poiints show random pattern
Determine the proportion of variance in salary that the number of years spent modelling uniquely explains once the models’ age was taken into account:
Hierarchical regression
A 2.0%
b17.8%
c39.7%
d42.2%
A –>The R square change in step 2 was .020,
Test for multicollinearity (select tolerance and VIF statistics).
Based on this information, what can you conclude about the suitability of your regression model?
AThe VIF statistic is above 10 and the tolerance statistic is below 0.2, indicating that there is no multicolinearity.
BThe VIF statistic is above 10 and the tolerance statistic is below 0.2, indicating that there is a potential problem withmulticolinearity.
CThe VIF statistic is below 10 and the tolerance statistic is above 0.2, indicating that there is nomulticolinearity.
DThe VIF statistic is below 10 and the tolerance statistic is above 0.2, indicating that there is a potential problem withmulticolinearity.
B
The following graph shows:
A. Regression assumptions met
B. Non-linearity = could indicate curve
C. Hetrodasticity + Non-linearity
D. hetrodasticity
A
A consumer researcher was interested in what factors influence people’s fear responses to horror films. She measured gender (0 = female, 1 = male) and how much a person is prone to believe in things that are not real (fantasy proneness) on a scale from 0 to 4 (0 = not at all fantasy prone, 4 = very fantasy prone). Fear responses were measured on a scale from 0 (not at all scared) to 15 (the most scared I have ever felt). What is the likely population value of the parameter describing the relationship between gender and fear?
Somewhere between 3.369 and 0.517
