Week 4: Multiple Regression Flashcards
What is the decision tree for multiple regression? - (4)
- Continous
- Two or more predictors that are continous
- Multiple regression
- Meets assumptions of parametric tests
simple linear regression
the outcome variable Y is
predicted using the equation of a straight line
Multiple regression still uses the same basic equation of …. but the model is still complex
Multiple regression is the same as simple linear regression expect for - (2)
every extra predictor you include, you have to add a coefficient;
so, each predictor variable has its own coefficient, and the outcome variable is predicted from a combination of all the variables multiplied by their respective coefficients plus a residual term
Multiple regression equation
In multiple regression equation, list all the terms - (5)
- Y is the outcome variable,
- b1 is the coefficient of the first predictor (X1),
- b2 is the coefficient of the second predictor (X2),
- bn is the coefficient of the nth predictor (Xn),
- εi is the difference between the predicted and the observed value of Y for the ith participant.
Multiple regression uses the same principle as linear regression in a way that
we seek to find the linear combination of predictors that correlate maximally with the outcome variable.
Regression is a way of predicting things that you have not measured by predicting
an outcome variable from one or more predictor variables
Regression can be used to produce a
linear model of the relationship between 2 variables
Record company interested in creating model of predicting recording sales from advertising budget and plays on radio per week (airplay)
- Example of it’s MR plotted on + number of vars measured, what vertical axis shows, horizontal and third axis shows - (4)
It is a three dimensional scatter plots, which means there are three axes measuring the value of the three variables.
The vertical axis measures the outcome, which in this case is the number of album sales.
The horizontal axis measures how often the album is played on the radio per week.
The third axis, which can can think of being directed into the page measures the advertising budget.
Can’t plot a 3D plot of MR as shown here
for more than 2 predictor (X) variables
The overlap in the diagram is the shared variance, which we call the
covariance
covariance is also referred to as the variance
shared between the predictor and outcome variable.
What is shown in E?
The variance in Album Sales not shared by the predictors
What is shown in D?
Unique variance shared between Ad Budget and Plays
What is shown in C?
The variance in Album Sales shared by Ad Budget and Plays
What is shown in B?
Unique variance shared between Plays and Album Sales
What is shown in A?
Unique variance shared between Ad Budget and Album Sales
If you got two prediictors thart overlap and correlate a lot then it is a .. model
bad model can’t uniquely explain the outcome
In Hierarchical regression, we are seeing whether
one model explains significantly more variance than the other
In hierarchical regression predictors are selected based on
past work and the experimenter
decides in which order to enter the predictors into the model
As a general rule for hierarchical regression, - (3)
known predictors (from other research) should be entered into the model first in order of their importance in predicting the outcome.
After known predictors have been entered, the
experimenter can add any new predictors into the model.
New predictors can be entered either all in one go, in a stepwise manner, or hierarchically (such that the new predictor
suspected to be the most important is entered first).
Example of hierarchical regression in terms of album sales - (2)
The first model allows all the shared variance between Ad budget and Album sales to be accounted for.
The second model then only has the option to explain more variance by the unique contribution from the added predictor Plays on the radio.
What is forced entry MR?
method in which all predictors are forced
into the model simultaneously.
Like HR, forced entry MR relies on
good theoretical reasons for including the chosen predictors,
Different from HR, forced entry MR
makes no decision about the order in which variables are entered.
Some researchers believe that about forced entry MR that
this method is the only appropriate method for theory testing because stepwise techniques are influenced by random variation in the data and so rarely give replicable results if the model is retested.
How to do forced entry MR in SPSS? - (4)
Analyse –> Linear –> Regression
Put outcome in DV and IVs (predictors, x) in IV box
Can select a range of statistics in statistics box and press okay to check colinearity assumption
Can also click plots to check assumptions of homoscedasticity and lineartiy
Why select colinearity diagnostics in statistics box for regression? - (2)
This option is for obtaining collinearity statistics such as the
VIF, tolerance,
Checking assumption of no multicolinearity
Multicollinearity exists when there is a
strong correlation between two or more predictors in a regression model.
Multicollinearity poses a problem only for multiple regression because
simple regression requires only one predictor.
Perfect collinearity exists when at least
e.g., two predictors are perfectly correlated , have a correlation coefficient of 1
If there is perfect collinearity between predictors it
becomes impossible
to obtain unique estimates of the regression coefficients because there are an infinite number of combinations of coefficients that would work equally well.
Good news is perfect colinearity is rare in
real-life data
If two predictors are perfectly correlated then the values of b for each variable are
interchangable
The bad news is that less than perfect collinearity is virtually
unavoidable
As colinearity increases, there are 3 problems that arise - (3)
- Untrustory bs
- Limit size of R
- Importance of predictors
As colinearity increases, there are 3 problems that arise
Untrustworthy bs - (3)
As collinearity increases so do the standard errors of the b coefficients.
big standard errors for b coefficients means that these bs are more variable across samples
b coefficient in our sample is less likely to represent the population.
As colinearity increases, there are 3 problems that arise
Limit size of R - (2)
two predictors are highly correlated - the second predictor accounts for same variance accounted for first variable –> the second predictor accounts for a very little unique variance
If two predictors completely uncorrelated then second predictor likely to account for different varinace in outcome then acocunted for first predictor
As colinearity increases, there are 3 problems that arise
importance of predictors - (3)
Multicollinearity between predictors makes it difficult
to assess the individual importance of a predictor.
If the predictors are highly correlated, and each accounts for similar variance in the outcome, then how can we know
which of the two variables is important?
Quite simply we can’t tell which variable is important – the model could include either one, interchangeably.
One way of identifying multicollinearity is to scan a
a correlation matrix of all of the predictor
variables and see if any correlate very highly (by very highly I mean correlations of above .80
or .90)
SPSS produces colinearity diagnoistics which is - (2)
variance inflation factor (VIF) and tolerance
The VIF indicates whether a
predictor has a strong linear relationship with the other predictor(s).
If VIF statistic is above 10 there is a good reason to worry about
potential problem of multicolinearity
If VIF statistic above 10 or approaching 10 then what you would want to do is have a - (2)
look at your variables to see if you need to include all variables whether all need to go in model
if high correlation between 2 predictors (measuring same thing) then decide whether its important to include both vars or take one out and simplify regression model
Related to the VIF is the tolerance
statistic, which is its
reciporal (1/VIF) = inverse of VIF
In tolerance, value below 0.2 shows
issue with multicolinerity
In Plots in SPSS, you put - (2)
ZRESID on Y and ZPRED on X
Plot of residuals against predicted to asses homoscedasticity
What is ZPRED? - (2)
(the standardized predicted values of the dependent variable based on the model).
These values are standardized forms of the values predicted by the model.
What is ZRESID? - (2)
(the standardized residuals, or errors).
These values are the standardized differences between the observed data and the values that the model predicts).
A plot of SRESID (studentised residuals) on y axis and ZPRED on x axis will show up any
heteroscedasticity also
SPSS in multiple linear regression gives descriptive outcoems which is - (2)
- basics means and also a table of correlations between variables.
- This is a first opportunity to determine whether there is high correlation between predictors, otherwise known as multi-collinearity
SPSS also gives summary of overall model for example whether model is successful in predicting
record sales
In model summary of SPSS, it captures how the model or models explain
variance in terms of R squared, and more importantly how R squared changes between models and whether those changes are significant.
Diagram of model summary
What is the measure of R^2?
measure of how much of the variability in the outcome is accounted for
by the predictors
The adjusted R^2 gives us an estimate of
fit in the general population
The Durbin-Watson statistic if specificed tells us whether the - (2)
assumption of independent errors is tenable (value less than 1 or greater than 3 raise alarm bells)
value closer to 2 the better = assumption met
SPSS output for MR = ANOVA table which performs
F-tests for each model
SPSS output for MR contains ANOVA that tests whether the model is
significantly beter at predicting the outcome than using the mean as a ‘best guess’
The F-ratio represents the ratio of
improvement in prediction that results from fitting the model, relative to the inaccuracy that still exists in the model
We are told the sum of squares for model (SSM) - regression line in output which represents
improvement in prediction resulting from fitting a regression line to the data rather than using the mean as an estimate of the outcome
We are told residual sum of squares (Residual line) in this output which represents
total difference between
the model and the observed data
DF for Sum of squares Model for regression line is equal to
number of predictors (e.g., 1 for first model, 3 for second)
DF for Sum of Squares Residual for MR is - (2)
Number of observations (N) minus number of coefficients in regression model
(e.g., M1 has 2 coefficents - one for predictor and one for constant, M2 has 4 - one for each 3 predictor and one for constant)
The average sum of squares in ANOVA table is calculated by
calculated for each term (SSM, SSR) by dividing the SS by the df. T
How is the F ratio calculated in this ANOVA table?
F-ratio is calculated by dividing the average improvement in prediction by the model (MSM) by the average
difference between the model and the observed data (MSR)
If the improvement due to fitting the regression model is much greater than the inaccuracy within the model then value of F will be
greater than 1 and SPSS calculates exact prob (p-value) of obtaining value of F by change
What happens if b values are positive?
there is a positive relationship between the predictor and the outcome,
What happens if the b value is negative?
represents a negative relationship between predictor and outcome variable?
What do the b values in this table tell us what relationships between predictor and outcome variable? (3)
Indicating positive relationships so as advertising budget increases, record sales increases (outcome)
plays on ratio increase as do record sales
attractiveness of band increases record sales
The b-values also tell us, in addition to direction of relationship (pos/neg) , to what degree each
predictor affects the outcome if the effects of all other predictors are held constant:
B-values tell us to what degree each predictor affects the outcome if the effects of all other predictors held constant
e.g., advertising budget - (3)
(b = 0.085):
This value indicates that as advertising budget (x)
increases by one unit, record sales (outcome, y) increase by 0.085 units.
This interpretation is true only if the
effects of attractiveness of the band and airplay are held constant.
Standardised versions of b-values are much more easier to interpret as
not dependent on the units of measurements of variables
The standardised beta values tell us that
the number of standard deviations that the outcome will change as a result of one standard deviation change
in the predictor.
The standardized beta values are all measured in standard deviation
units and so are directly comparable: therefore, they provide a
a better insight into the
‘importance’ of a predictor in the mode
If two predictor variables (e.g., advertising budget and airplay) have virtually identical standardised beta values (0.512, and 0.511) it shows that
both variables have a comparable degree of importance in the model
The confidence intervals of unstandardised beta values are boundaries constructed such that
95% of these sampels these boundaries containn true value of b
A good regression model will have a narrow and small CI interval indicating
value of b in this sample is close to the true value of b in the populatio
A bad regression model have CI that cross zero indicating that
in some samples the predictor has a negative
relationship to the outcome whereas in others it has a positive relationship
In image below, which are the two best predictors based on CIs and one that isn’t as (2)
two best predictors (advertising and airplay) have very tight confidence intervals indicating that the estimates for the current model are likely to be representative of the true population values
interval for attractiveness is wider (but still does not cross zero) indicating that the parameter for this variable is less representative, but nevertheless significant.
If you do part and partial correlations in descriptive box, there will be another coefficients table which looks this like:
The zero-order correlations are the simple
Pearson’s correlation coefficients
The partial correlations represent the
represent the relationships between each predictor and the outcome variable, controlling for the effects of the other two predictors.
The part correlations - (2)
represent the relationship between each predictor and the outcome, controlling for the effect that the other two variables have on the outcome.
representing the unique relationship each predictor has with otucome
In this table , zero-order correlation is calculated by - (2)
variance of outcome explained by predictors divided by total
(A+C)/(A+B+C+E)
Partial correlations in example is calculated by - (2)
unique variance in outcome (ignore all other predictors) explained by predictor divided by variance in outcome not explained by all other predictors
A/A+E
Part correlations are calculated by - (2)
unique variance in outcome explained by predictor divided by total variance in outcome
A/A+B+C+E
How to interpret this image in terms of colinearity - VIF and tolerance
For our current model the VIF values are all well below 10 and the tolerance statistics all well above 0.2;
therefore, we can safely conclude that there is no collinearity within our data.
We can produce casewise diagnostics to see a (2)
summary of residuals statistics to be examined of extreme cases
To see whether individual scores (cases) influence the modelling of data too much
SPSS casewise diagnostics shows cases that have a standardised residuals that are (2)
less than -2 or greater than 2
(We expect about 5% of our cases to do tha and 95% to have standardised residuals within about +/- 2.)
What does this casewise diagnostic show? - (2)
- 99% of cases should lie within ±2.5 so expect 1% of cases lie outside limits
- From cases listed, clear two cases (1%) lie outside of limits (case, 164 [investigate further has residual 3] and 179) - 1% which isconform to accurate model
If there are many more cases we likely have (more than 5% of sample size) in case wise then
broken the assumptions of the regression
If cases are a large number of standard deviations from the mean, we may want to in casewise diagnostics
investigate and potentially remove them because they are ‘outliers’
Diagram of assumption of homoscedasticity and linearity of ZRESID againsr ZPRED
What does this partial plot show? - (2)
the partial plot shows the strong positive relationship to album sales.
There are no obvious outliers and the cloud of dots is evenly spaced out around the line, indicating homoscedasticity.
What does this plot show? (2)
the plot again shows a positive relationship to album sales, but the dots show funnelling,
There are no obvious outliers on this plot, but the funnel-shaped cloud indicates a violation of the assumption of homoscedasticity.
P plot and histogram of normally distributed
P plot for skewed distirbution histogram
If R^2 is 0.374 (outcome var in productivity and 3 predictors) then it shows that
37.4% of the variance in productivity scores was accounted for by 3 predictor variables
This table tells us in terms of standardised beta values that (outcome is productivity)
holidays had standardized beta coefficient of 0.031 whereas cake had a much higher standardized beta coefficient of 0.499 which tells us that amount of cake given out much better predictor of productivity than the amount of holidays taken
For pay we have a beta coefficient of 0.323 which tells us that pay was also a pretty good predictor in the model of productivity but slightly less than cake
What does this table tells us in terms of signifiance? - (3)
- P value for holidays is 0.891 which is not significant
- P value for cake is 0.032 is significant
- P value for pay is 0.012 is significant
What does this image show in terms of VIF?
o All below 10 here showing we are unlikely to have a problem with multicollinearity so we can not worry about that for this data
In ANOVA it is comparing M2 with all its predictor variables with
baseline not M1
To see if M2 is an improvement of M1 in HR we need to look at … in model summary
change statistics
What does this change statistic show in terms of M2 and M1
M2 explains an extra 7.5% which is sig
Each of these beta values shown in table has an associated standard error indicating to what extent and used to determine (2)
values would vary across different samples,
and these standard errors are used to determine
whether or not the b-value differs significantly from zero
For this output interpret whether predicotrs are sig predictors of record scales and magnitude t statistic on impact of record sales - (2)
For this model, the advertising budget (t(196) = 12.26, p < .001), the amount of radio play prior to release (t(196) = 12.12, p < .001) and attractiveness of the band (t(196) =4.55, p < .001) are all significant predictors of record sales.
From the magnitude of the t-statistics we can see that the advertising budget and radio play had a similar impact,
whereas the attractiveness of the band had less impact.
Diagram of outliers
All residuals should lie within ….. SDs for no outliers /normal amount of outliers
-3 and 3 SD
Which variables (if any) are highly correlated?
Weight, Activity, and the interaction between them are statistically significant
Diagram of normal, skewed to left (pos) and skewed to right (neg) of p-plots
Diagram of excluded variables table in SPSS
- Tells that OCD interpretiotn of intrustrions would have not have a significant impact on model’s ability to predict social axniety
When checking assumption fo regression, what does this graph tell you?
Normality of residuals
A consumer researcher was interested in what factors influence people’s fear responses to horror films. She measured gender and how much a person is prone to believe in things that are not real (fantasy proneness). Fear responses were measured too. In this table, what does the value 847.685 represent?
The residual error in the prediction of fear scores when both gender and fantasy proneness are included as predictors in the model.
A psychologist was interested in whether the amount of news people watch predicts how depressed they are. In this table, what does the value 3.030 represent?
The improvement in the prediction of depression by fitting the model
When checking the assumption of the regression, the following graph shows (hint look at axis titles)
Regression assumptions that have been met
A consumer researcher was interested in what factors influence people’s fear responses to horror films. She measured gender (0 = female, 1 = male) and how much a person is prone to believe in things that are not real (fantasy proneness) on a scale from 0 to 4 (0 = not at all fantasy prone, 4 = very fantasy prone). Fear responses were measured on a scale from 0 (not at all scared) to 15 (the most scared I have ever felt).
Based on the information from model 2 in the table, what is the likely population value of the parameter describing the relationship between gender and fear?
Somewhere between −3.369 and −0.517
A consumer researcher was interested in what factors influence people’s fear responses to horror films. She measured gender (0 = female, 1 = male) and how much a person is prone to believe in things that are not real (fantasy proneness) on a scale from 0 to 4 (0 = not at all fantasy prone, 4 = very fantasy prone). Fear responses were measured on a scale from 0 (not at all scared) to 15 (the most scared I have ever felt).
How much variance (as a percentage) in fear is shared by gender and fantasy proneness in the population?
13.5%
Recent research has shown that lecturers are among the most stressed workers. A researcher wanted to know exactly what it was about being a lecturer that created this stress and subsequent burnout. She recruited 75 lecturers and administered several questionnaires that measured: Burnout (high score = burnt out), Perceived Control (high score = low perceived control), Coping Ability (high score = low ability to cope with stress), Stress from Teaching (high score = teaching creates a lot of stress for the person), Stress from Research (high score = research creates a lot of stress for the person), and Stress from Providing Pastoral Care (high score = providing pastoral care creates a lot of stress for the person). The outcome of interest was burnout, and Cooper’s (1988) model of stress indicates that perceived control and coping style are important predictors of this variable. The remaining predictors were measured to see the unique contribution of different aspects of a lecturer’s work to their burnout.
Which of the predictor variables does not predict burnout?
Stress from research
Using the information from model 3, how would you interpret the beta value for ‘stress from teaching’?
As stress from teaching increases by one unit, burnout decreases by 0.36 of a unit.
How much variance in burnout does the final model explain for the sample?
80.3%
A psychologist was interested in predicting how depressed people are from the amount of news they watch. Based on the output, do you think the psychologist will end up with a model that can be generalized beyond the sample?
No, because the errors show heteroscedasticity.
The following graph shows:
A. regression assumption has been met
B. non linearity
C. Hetrodasticity
D. Hetrodasticity
A. Regression assumptions have been met
