Chapter 9: Linear Model (Regression)

Question 1

What is the difference between a linear model and correlation?

Answer 1

Linear model differs from that of a correlation only in that it uses an unstandardized measure of the relationship (b1) and a parameter (b0) that tells us the value of the outcome when the predictor is 0.

b0 = intercept & represents the value of the outcome when predictor is 0

Question 2

Any straight line can be defined by two things:

Answer 2

1) the slope (b1) - also known as the relationship between the predictor and the outcome in unstandardized units
2) the point at which the line crosses the vertical axis of the graph, known as the intercept (b0)

b1 & b0 are parameters known as regression coefficients

b1 represents mean change in the outcome for one unit change in predictor

Question 3

Regression is used for two things:

Answer 3

1) prediction
2) test theories/explanations

Question 4

A good model…

Answer 4

1) fits the data better than a model with no predictors
2) should account for an amount of variance that is judged to be of practical and/or scientific significance
3) individual predictors/regression coefficients are significantly different than 0
4) should not have any outliers biasing the model
5) expected to predict the outcomes well in other samples
6) can be generalized to additional samples (cross validation) because assumptions are met

does not prove causation, sees if data consistent w/ causal hypothesis

once a model is estimated, it can be used for forecasting, called predicted value

Question 5

Same intercept, different slopes

Answer 5

b0 (intercept) is the same in each but b1 (slope) is different. it looks like three lines coming out of the same point in the graph.

Question 6

Same slope, different intercept

Answer 6

b1(slope) is the same in each but b0(intercept) is different in each model. it looks like three separate lines going in the same direction, but do not connect at any point.

+ b1 = + relationship w/ outcome; - b1 = - relationship w/ outcome

the slope (b1) tells us what the model looks like (shape); the intercept (b0) locates the model in geometric space.

Question 7

What is a regression analysis?
What are the two types?

Answer 7

term for fitting a linear model to data and using it to predict values of an outcome (dependent) variable from one or more predictor (independent) variables.

simple regression: one predictor variable in the linear model
multiple regression: several predictors in the linear model

Question 8

What are residuals?

Answer 8

differences between what the linear model predicts and the observed data

Question 9

What is the residual sum of squares (SSr)?

Answer 9

gauge of how well a linear model fits the data. it represents the degree of inaccuracy when the best model is fitted to the data. if it is a large number, it means the model is not representative of the data (a lot of error in prediction). if it is a small number, the line is representative of the data.

total amount of error in a model/does not tell us how good the model is

higher with more people

Question 10

Ordinarly least squares (OLS) regression

Answer 10

method of regression in which the parameters of the model are estimated using the method of least squares
essentially, getting b values that make the sum of the squared residuals (error b/w observed and model) as small as possible

Question 11

What happens wheen there are more predictors?

Answer 11

expanded models with more predictors account for even more variance. unless the IVs are completely independent of one another, the predictors change as additional predictors are added

regression planes must be used to visualize.

w/ 3 or more, you cant visualize it.

Question 12

What is goodness-of-fit?

good models exist on a continuum

Answer 12

assesses how well the model fits the observed data. we do this because even though the model may be the best one available, it can still be a bad fit. usually based on how well the data predicted by the model corresponds to the data actually collected.
done by comparing the complex model against a baseline model to see whether it improves how well we can predict the outcome, then calculating the error. if the complex model is any good, it should have significantly less error than the simple model

simple model is usually the mean of the outcome

R^2 and F Statistic

Question 13

What is the total sum of squares (SSt)?

Answer 13

represents how good the mean is as a model of the observed outcome scores.
it is a measure of the total variability within a set of observation (total squared deviance between each observation and the overall mean of all observations)

Question 14

What is the model sum of squares (SSm)?

Answer 14

the improvement in prediction resulting from using the linear model rather than the mean. this difference shows us the reduction in the inaccuracy of the model resulting from fitting the regression model to the data.
if SSm is large, the linear model is very different from using the mean to predict the outcome variable. this implies that the linear model has made a big improvement in predicting the outcome.
if SSm is small, using the linear model is little better than using the mean

SSm = SSt (total sum of squares) - SSr (residual sum of squares)

higher with more predictors

Question 15

What is R^2?

Answer 15

represents the amount of variance in the outcome explained by the model (SSm) relative to how much variation there was to explain in the first place (SSt).
proportion of variation in the outcome that can be predicted from the model
indicates whether the model is of scientific and/or practical significance

R^2 = SSm / SSt

we want this number to be high - r will tell us overall fit of the regression model

Question 16

Mean Squares (MSm and MSr)

Answer 16

measure of average variability, based on the number of differences that were added up
MSm = SSm/k (df; number of predictors in the model)
MSr = SSr/ N - k - 1 (N = number of observations; k = number of parameters being estimates aka as all the b values)

dividing undoes the biasing effect of # of predictors

Question 17

What is the F-statistic?

Answer 17

measure of how much the model has improved the prediction of the outcome compared to the level of inaccuracy of the model.
if a model is good MSm will be large and MSr will be small
large F statistic = greater than 1 (1 indicated no improvement)

F = MSm/MSr

if associated p-value is less than .05, there is significant improvement in prediction over the baseline model w/ no predictors
as the F gets higher, p gets lower (H0 less tenable)

Question 18

What can the F statistic tell us about R^2?

Answer 18

F can be used to calculate the significance of R^2 (how different it is from 0)
F = (N - k - 1)R^2 / k(1-R^2)
N = # of cases/participants, k = number of predictors in the model

if associated p-value is less than .05, R^2 is significantly different from 0

Question 19

What is a flat model?

Answer 19

model in which the same predicted value arises from all values of the predictor values, and will have b-values of 0 for the predictors
if a variable significantly predicts an outcome, it should haev a b value that is different from 0

regression coeff of 0 means a unit change in the predictor results in no change in the predicted value of the outcome and the linear model is flat

Question 20

What is the t-statistic?

Answer 20

tests whether a b-value is significantly different than 0.
if the test is significant, we might interpret this as supporting a hypothesis that the b (regression coeffs) is significantly different (p < .05) from 0 and that the predictor variable contributes significantly to our ability to estimate values of the outcome, after accounting for the other predictors in the model. Each b gets its own test.

t = b(observed)-b(expected)/ SEb = b(observed)/ SEb

b(expected) = b value we expect to obtain if null is true, so 0.
b(observed) = b we calculate
SEb = how much error we estimate is likely in our b

when the SE is small even a small deciation from 0 can reflect a sig difference because b is representative of the majority of possible samples

Question 21

What is generalization?

Answer 21

ability of a model to be applied to other samples aside from the one which it was based on. if it is not generalizable, then we must restrict conclusions only to the sample used.

Question 22

What are outliers?

Answer 22

data with extreme value on the outcome variable, Y (large residuals)
case that differs substantially from the main data trend. it can affect the estimates of the regression coefficients. outliers can be assessed by unstandardized, standardized, and studentized residuals. they are NOT always influential

Question 23

unstandardized residuals

Answer 23

raw differences between the predicted and observed values of the outcome variable

Question 24

standardized residuals

Answer 24

residuals converted to z scores/represented in SD units
1) if they are greater than 3, then they are cause for concern because a value that high is unlikely to occur
2) if more than 1% of our sample cases have residuals w an absolute value greater than 2.5 there is evidence that the level of error in the model is unacceptable
3) if more than 5% of cases have residuals with an absolute value greater than 2, then the model may be a poor representation of the data

use this bc its easier to interpret

Question 25

studentized residuals

Answer 25

unstandardized residual divided by an estimate of its SD that varies point by point
have the same properties as standardized, but provide a more precise estimate of the error variance of a special case

use this bc its easier to interpret

Question 26

adjusted predicted value

Answer 26

predicted value of the outcome for that case from a model in which the case is excluded
estimate the model parameters excluding a particular case and use this new model to predict the outcome for the case that was excluded. if the model is stable then the predicted value of a case should be the same regardless of whether the case was used to estimate the model

Question 27

deleted residual

Answer 27

difference between the adjusted predicted value and the original observed value

tell us about the influence of cases on the ability of the model to predict that case, but not about how the case influences the whole model

Question 28

studentized deleted residual

Answer 28

the deleted residual is divided by the standard error to give us this value. it can then be compared across different regression analyses

Question 29

Leverage

Answer 29

gauges the influence of the observed value of the outcome over the predicted values.

Question 30

high leverage points

Answer 30

data with an extreme value on a predictor variable (x)
these points are extreme on the x-axis. if it drags the regression line towards it, it is influential

all leverage points should be close to the average value (k+1)/n
cases with 2-3x the average leverage value should be investigated/concerning

range from 0-1. a value of 1 indicates the case has complete influence over prediction

Question 31

When is a case influential?

Answer 31

a case is influential if the model parameter estimates change substantially if the case is deleted and the model reestimated
a good model should not be so fragile that 1-2 cases change it a lot
cases will be influential if: they have some combination of being extreme on X (leverage) and extreme on Y (outliers)
influence is what matters most
if conclustions change based on 1-2 data points, conclusions are said to be fragile

not all influential points have large residuals

can be examined using: Cook’s distance, Difference in Beta (DFBeta), Difference in Fit (DFFit).

Question 32

Cook’s distance

Answer 32

measure of the overall influence of a case on the model
abs values greater than 1 may be a concern

Question 33

Difference in Beta (DFBeta)

Standardized

Answer 33

measure of how much the estimates of the b’s change when a case is deleted
abs values greater than 1 may be a concern

Question 34

Difference in Fit (DFFit)

Standardized

Answer 34

measure of the difference in prediction when a case is deleted
abs values greater than 1 may be a concern

Question 35

Consider deleting high leverage points or outliers only when:

Answer 35

they are influential
first check that influential points are not coding error
if it is not coding error: does the case change the conclusions? is it possible to get more observations near that value of X?
if case does change the conclusion: report the results with and without the influential case or restrict your analysis to values of X where the relationship holds
do not use this to drop cases to create desired results (phack)

Question 36

Assumptions of the linear model

Answer 36

1) additivity and linearity
2) independent errors
3) homoscedasticity
4) normally distributed error

Question 37

additivity & linearity

Answer 37

outcome variable and predictors are linearly related/can be described by a linear model
do not use a linear model to describe a nonlinear relationship

Question 38

independent errors

Answer 38

residual terms should be uncorrelated/independent
assumption necessary for CIs and significance tests to be valid
if violated, use robust methods or a multilevel model

Question 39

homoscedasticity

Answer 39

residuals at each level of the predictor should have the same variance
if violated, CIs and significance tests are invalidated. use weighted least squares regression instead.

Question 40

normally distributed errors

Answer 40

residuals in the model are random, normally distributed variables with a mean of 0
does not matter with large sample sized because of CLT
if violated with a small sample size, use bootstrapped CIs

Question 41

other assumptions/considerations of the linear model

Answer 41

1) predictors are uncorrelated with external variables
2) Variable type: quantitative or dichotomous predictors, and continuous unbounded criterion
3) no perfect mullticollinearity
4) non zero variance

Question 42

predictors are uncorrelated with external variables

Answer 42

should be no external variables that correlate with any of the variables included in the model
regression results can be biased by an omitted (3rd) variable
if violated, conclusions are unreliable

Question 43

Variable type: quantitative or dichotomous predictors, and continuous unbounded criterion

Answer 43

all predictor variables must be quanititative or categorical, and the outcome must be quantitative, continuous, and unbounded

Question 44

no perfect multicollinearity

Answer 44

if your model has more than one predictor then there should be no perfect linear relationship between 2 or more of the predictors (predictors should not correlate too highly)
if violated, lead to untrustworthy estimates of the b’s, and SE gets very big

Question 45

non zero variance

Answer 45

predictors should have some variation in value (not have variance of 0)

Question 46

cross validation of the model

Answer 46

assessing the accuracy of a model across different samples (how it generalizes to different samples)
two main methods: adjusted R^2 and data splitting

Question 47

adjusted R^2

Answer 47

tells us how much variance in the predicted outcome would be accounted for if the model had been derived from the population from which the sample was taken (estimates what r^2 would be in a new sample)
when you try to apply a model to a different sample, R^2 from sample 1 to sample 2 will drop, causing a loss of predictive power known as SHRINKAGE.
shrinkage occurs because the process of fitting a model capitalizes on chance

more capitalizing on change + more shrinkage will occur when there are more predictors and with smaller sample sizes.
with large samples, a model will be well estimated and shrinkage will be minimal

regression models are optimized for the sample they were created from

Question 48

data splitting

Answer 48

randomly splitting your sample data, estimating the model in both halves of the data, and comparing the resulting models (80/20(model))

Question 49

To assume b’s are normally distibuted, we need a large sample. How big does it have to be?

Answer 49

situation specific power analysis is the best way to determine sample size needed.
General guidelines:
- if you expect to find a large effect: 80 people or higher
- if you expect to find a medium effect: 100 people or higher if there are 6 predictors or less
- if you expect to find a small effect: don’t bother unless you can get a very large sample

Question 50

Multiple regression: methods of entering predictors into a model

Answer 50

1) forced: predictors are added into the model at once. it is useful for testing theory. no established predictors
2)** hierarchical:** predictors added in blocks. established predictors added earlier in the process. new predictors asses as a group last. useful for testing theory or validity of new predictors.
3) stepwise: automatic method. predictors added to the model one by one based on partial correlations. process stops when removal criterion is met (i.e., regression coeff is not significant for the added predictor). useful for exploratory analyses when you have no idea what’s going on what want to generate hypotheses. it gives models that can’t generalize, and is frowned upon.
- this method is sensitive to sampling variation and the results don’t generalize

adding control variables into the model doesn’t purify the analysis and their inclusion can result in inappropriate inferences

Question 51

Parsimony

Answer 51

• explaining data while being as simple as possible
• accounting for variance in the simplest way
• more predictors account for more variance
• R^2 increases with more predictors added/see if new predictors have value
• change R^2 from simple model to complex model: if its sig, then sampling error alone cannot account for this difference.

Question 52

Assessing parsimony

Answer 52

• change in R^2 (significance and magnitude)
• Akaike Information Criterion (AIC): lowest AIC = most parsimonious model, penalizes you for adding predictors

you can compare different models and access parsimony using R^2 change and AIC.
R^2 should significantly change, magnitude change, lower AIC = argument for the complex model

Question 53

What is multicollinearity?

multiple regression

Answer 53

• occurs when the predictor variables themselves are related to each other (highly correlated)
• for ex. trying to predict lawyer salary based on age and experience (hard to tease apart)
• level of multicollinearity varies from none to perfect multicollinearity (continuum). if none, it means that all predictor variables are unrelated. if perfect, can’t fit a regression model because there are infinite models
• mild multicollinearity is not a big deal but high multicollinearity makes it more difficult to estimate the b’s
• high multicollinearity can result in the standard errors of the coeffs being very high. this means more errors in each estimate and a wider sampling distribution (parameters not well estimated)
• parameter estimates change wildly from sample to sample; CI’s will be wide; and harder to find significance

Question 54

How to diagnose multicollinearity?

Answer 54

• simplest way: examine the correlations among predictors. correlations that are higher than .8 or .9 suggest multicollinearity may be an issue
• examining the correlation matrix will miss more subtle forms of multicollinearity
• we need stats that are specific to detecting multicollinearity. one is the variance inflation factor (VIF) for each predictor. (VIF = 1/(1-R^2k)
• VIF > 10 is a problem

Question 55

how to deal with multicollinearity?

Answer 55

1) do nothing
2) get rid of the variables
3) combine the correlated variables
4) use a method that can handle highly correlated variables like partial least squares or principal component analysis

Question 56

how can the unstandardized simple regression equation be written in standard form?

Answer 56

• variables expressed as z scores
• Zy = r(Zx)
• predicting Y’s z score, input is not the person’s score on X but rather their z score on X

Question 57

how can the unstandardized multiple regression equation be written in standard form for 2 predictors?

Answer 57

Zy = Beta(Zx) + Beta(Zx) …
- the betas take place of the correlation coeffs becaue we have to account for the other predictors
- the std multiple regression equation works the same way as the std simple regression equation. Plug in someone’s z score on the X variables to get their predicted z score on the Y variable
- the std betas refer to how many SDs the outcome will change for every SD change in the predictor
- can be extended to include more predictors

std betas help assess relative importance of dif predictors in SD units