Logistic Regression

Question 1

what is logistic regression (LR) for?

Answer 1

Logistic regression is an example of a non-linear regression model, which is what we need when we have a dichotomous or categorical DV

Question 2

the assumptions is LR are characteristically ….

Answer 2

less severe – relatively assumption free

Question 3

What are the 3 main reasons for performing a logistic regression rather than a standard multiple regression?

Answer 3

1) DV is categorical, and therefore 2) Line of best fit will be sigmoidal, not linear, and as such 3) There will be non-normality and heteroscedasticity in the residuals if OLS regression is used, which violates important assumptions of this method

Question 4

how does LR build a model?

Answer 4

by measuring the deviance of predictors, and including them or excluding them based on their contribution to predicting the outcome variable …. LR says: Does an individual predictor increase or decrease the probability of an outcome?

Question 5

as apposed to MR… LR uses a dichotomous DV, and …

Answer 5

continuous IVs -

Question 6

LR is also not…

Answer 6

linear,

Question 7

the predictive model is called XX

Answer 7

P hat

Question 8

the residuals are not…..

Answer 8

Residuals are clearly not normal (skewed)

Question 9

and exhibit …

Answer 9

heteroscedasticity – residuals are all or nothing, and not evenly distributed.

Question 10

so, Instead of the model fit being linear (which excludes the possibility of using probability, as a linear line can extend past 0 and 1, where probability lies), LOG_REG uses

Answer 10

a non-linear (sigmoidal) line of best ft.

Question 11

Probability means =

Answer 11

0-1 or a % 0-100) – likelihood of an event occurring

Question 12

Odds mean =

Answer 12

Odds = Probability of event divided by its component 1-the probability of an event

Question 13

why does LR use odds?

Answer 13

unpacks the maths nicely - actually it is converted back t p value after using odds

Question 14

in LR, instead of using the odds (which are xx), we use the …

Answer 14

asymmetric natural log

Question 15

what is the natural log called in LR

Answer 15

the logit

Question 16

what does the odds ratio mean in LR?

Answer 16

Odds ratio: relationship between the odds of an event occurring across levels of another variable (by how much do the odds of Y change as X increases by 1 unit?)

Question 17

and what does the ‘ratio of ratios’ mean?

Answer 17

Ratio of a ratios – the event of an event occurring as a function of levels of another variable. (e.g. odds of males having a disease with the odds of females having disease – i.e. the odds of these odds combined)

Question 18

why do we present the results in terms of log odds and odds ratio?

Answer 18

as it turns a non-linear relationship into the familiar linear one

Question 19

this enables us to subsequently …

Answer 19

test whether this coefficient is significantly different from 0 – just like a t-test in MR

Question 20

the predicted odds range from ?

Answer 20

0 to + ∞

Question 21

so when p>.50

Answer 21

odds>1 (.50 would be even at 1)

Question 22

the predicted odds varies ….

Answer 22

varies exponentially with the predictor(s)

Question 23

in comparison the natural logit ranges…..

Answer 23

from - ∞ to + ∞

Question 24

it reflects odds of being a case but

Answer 24

varies linearly with the predictor(s)

Question 25

the issue with this is

Answer 25

not very interpretable; – if p=.8; odds=4; logit=1.386

Question 26

The typical partial regression coefficients (B) indicate ….

Answer 26

increment in the logit given unit increment in predictor

Question 27

whereas the Odds ratios (eB) indcates?

Answer 27

the amount by which odds of being-a-case are multiplied given a unit increment in predictor (or change in level of predictor if predictor is categorical)

Question 28

if MR uses OLS…. LR uses…..

Answer 28

Ð maximum likelihood estimation; an ITERATIVE solution where the regression coefficients are estimated by trial-and-error and gradual adjustment. Ð (this seeks to maximize the likelihood (L) of the observed values of Y given a model and using the observed values of the predictors)

Question 29

if OLS uses the sum of squares….. LR uses….

Answer 29

uses measures of deviance rather than sums of squares Focus is lack of fit more focused on (1-R2) minimizing this – just the same as MR but flipped round Null Deviance, Dnull; similar to SSTotal = reflects amount of variability in data, the amount of deviance that could potentially be accounted for. Model Deviance, DK; similar to SSResidual = reflects amount of variability in data after accounting for prediction from k predictors

Question 30

For each model, a x x value is calculated – which is analogous to a F ratio etc for the overall model.

Answer 30

log likelihood

Question 31

LR uses xxx models

Answer 31

nested So, minimising the lack of fit of the model – maximising the likelihood of the data So, take models, compare sets of models with one another. Simplest way is comparing model with all the variables in, with no model at all..and compare subsets of the model, with and without individual predictors + sig of each predictor. (COMPARING 2 MODELS, ONE IS BIGGER AND ONE IS SMALLER AND NESTED IN THE BIGGER MODEL, COMPARING HIERARCHICALLY)

Question 32

If the xxx model is true then the LRT statistic is distributed as xxx with m df

Answer 32

Ð If the smaller model is true then the Likelihood ratio test (LRT) statistic is distributed as χ2 with m df if smaller model is correct, under assumptions, then LRT is chi 2 with m df. SOOOO its testing whether it is worth having those m parameters in the model, if the LRT is not any bigger under the chi 2 dist with m df then it is not worth putting the additional parameters in, and we prefer the simpler model – more parsimonious explanation – only prefer bigger model if it improved fit.

Question 33

this LR standard approach is more xxxxx than standard MR – resembles xxxxx MR.

Answer 33

Standard approach is more hierarchical than standard MR – resembles hierarchical MR. We only accept more predictors if they significantly enhance the degree of fit.

Question 34

TAKE HOME MESSAGE: Collectively when we have assessed our model, and the 3 predictors don’t enhance our model prediction. Then you ask…

Answer 34

would one model on its own be a better fit etc?

Question 35

a limitation of LR is that it is a xxxxxxx proceedure

Answer 35

low power

Question 36

how is it low power?

Answer 36

as catagorical - IVs are either 0 or a 1 needs big sample sizes

Question 37

what is Pseudo R2 ?

Answer 37

waste of time – analgous to R2 – McFaddons/ Cox n Snell / Nagelkerke – all crap. Not “variance accounted for” as not homoscedastic.

Question 38

how do you calculate DF for catagorical DVs?

Answer 38

To calculate DF for a binary DV (has disease: yes vs no)– you need to add up all the main effects and interactions. It is not N-1 as when DV is continuous. Similarly, categorical IVs need (m-1)*(n-1) parameters to capture the effects when there are m levels of the IV and n levels of the DV.

Question 39

what is the Wald statistic?

Answer 39

Wald statistic – a Z statistic, tested against chi2 again– similar to B or beta in OLS MR. Sometimes too conservative. This Wald in combination with the change in the model when you take that predictor out.

Question 40

just a note: HE mentions – this below would be in the exam WHEN VIEWING THE SPSS OUTPUT IN THE EXAM – I WOULD BE CAREFUL TO CHECK THE CODING OF THE VARIABLES – AS THEY PROBABLY WLL TRY CATCH US OUT. SAY HAVING AUTISM IS CODED AS 0, AND NOT HAVING AUTISM IS 1. YOU THINK OTHER WAY AROUND ETC. – see slide 39

Answer 40

HE mentions – this below would be in the exam WHEN VIEWING THE SPSS OUTPUT IN THE EXAM – I WOULD BE CAREFUL TO CHECK THE CODING OF THE VARIABLES – AS THEY PROBABLY WLL TRY CATCH US OUT. SAY HAVING AUTISM IS CODED AS 0, AND NOT HAVING AUTISM IS 1. YOU THINK OTHER WAY AROUND ETC. – see slide 39

Question 41

What effects are possible to examine in LR?

Answer 41

Possible to also examine – Full factorial (main effects and interactions between factors; no interactions involving covariates – like omnibus, doesn’t tell us much) – Complete: main effects and all interactions, including interactions with covariates (good) – Saturated: as complete only covariates treated as factors (just everything, huge model, granular way of looking at it – just IS the data itself)

Question 42

Common technique in LOG_REG is xxxxx

Answer 42

Common technique in LOG_REG is backward-stepwise: (like we would rerun without maternal warmth) Ð Iterative process (see SPSS class): Ð Begin with complete model Ð Remove non-significant variables Ð Re-run model and compare fit

Question 43

what is linearity of the logit?

Answer 43

Absence of multicollinearity among the predictors

Question 44

what are the two methods mentioned in slides to test multicollinearity?

Answer 44

HOSMER & LEMESHOW method BOX-TIDWELL

Question 45

the HOSMER & LEMESHOW method…..

Answer 45

Turn covariates into quartiles and enter as factor. Then compare the quartiles (do the EXP_Bs (odds ratio) increases at roughly equal levels/roughly linear trend)

Question 46

the BOX-TIDWELL method……?

Answer 46

In this approach, terms, composed of interactions between each predictor and its natural logarithm, are added to the logistic regression model. The assumption is violated if one or more of the added interaction terms are statistically significant. Construct a new predictor (in his e.g. Leadership*log(leadership) If this extra predictor is sig. then there is evidence of non-linearity in the logit

Question 47

Terminology in Multinomial LR: – Categorical predictors are called ‘x’ – Continuous predictors are called ‘x’

Answer 47

– Categorical predictors are called ‘factors’ – Continuous predictors are called ‘covariates’

Question 48

A number of problems may occur when

Answer 48

there are too few cases relative to the number of predictor variables.

Question 49

Logistic regression, like all varieties of multiple regression, is sensitive to extremely high correlations among predictor variables, how to test for both discrete and continuous variables?

Answer 49

To find a source of multicollinearity among the discrete predictors, use multiway frequency analysis (cf. Chapter 16) to find very strong relationships among them. To find a source of multicollinearity among the continuous predictors, replace the discrete predictors with dichotomous dummy variables and then use the procedures of Section 4.1.7. Delete one or more redundant variables from the model to eliminate multicollinearity.

Question 50

if the design is repeated measures, say the levels of the outcome variable are formed by the time period in which measurements are taken (before and after some treatment), then there is an issue with…..

Answer 50

Independence of Errors

Question 51

one way to remedy Independence of Errors?

Answer 51

One remedy is to do multilevel modeling with a categorical DV in which such dependencies are considered part of the model

Question 52

The model fitting information table includes information about the fit of two models - what are they?

Answer 52

: one is a model with no effects just the intercept (intercept only model) and the other (final model) is the model specified for this stage

Question 53

The -2 * log likelihood (-2LL) values for each model is given in the table with larger values indicating

Answer 53

indicating worse fitting models.

Question 54

The difference between -2LL values for two models is a …

Answer 54

likelihood ratio test statistic

and this is distributed approximately as the chi-squared distribution

Question 55

with degrees of freedom (df) equal to

Answer 55

the difference in number of parameters between the two models.

Question 56

if the statistic is highly significant for x df it indicates that

Answer 56

that there is be a statistically significant deterioration in fit from the final model to the intercept only model

Question 57

this means that some or all the parameters in the final model are useful in

Answer 57

explaining variance in outcome (i.e., DV category membership

Question 58

A good answer might also explain why there are x df. - explain how we calculate df in LOG_REG

Answer 58

I think this is how many levels of DV -1 for each parameter (i.e. main effect and interaction term) included here i ref the 2002 model answer question 2 “• The final model contains terms for age covariate effect, which for a 3-category DV requires 2 parameters; the gender factor also requires 2 parameters; and the age*gender interaction also requires 2 parameters. This explains the 6 df (=2+2+2). Note that effects in logistic regression are really effect*DV interactions, explaining why the 3 levels of the DV are relevant to the number of parameters needed. “

Question 59

a goodness-of-fit table compares …

Answer 59

the deterioration of fit between a saturated model (i.e., a model with 0 df, that provides the best possible fit to the data) and the final model for that stage of the analysis.

Nominal Log_Reg (from past exam)

Question 60

The two statistics in a goodness-of-fit table are?

Answer 60

deviance and pearson calculate the goodness of fit statistic in slightly different ways (deviance is a log likelihood ratio test).

Question 61

if the significance values are not significant it shows us…..

Answer 61

This shows that the final model for stage 1 is not a significantly worse fit to the data than a saturated model with x more parameters (x more parameters in the saturated model explain why these test statistics have 8 df). Thus the extra parameters in the saturated model are not particularly useful in fitting the data. A really good answer might explain why there are 8 more parameters required for the saturated model than for the final model of stage 1. But i dont really understad this tbh

Question 62

what information does the likelihood ratio tests table provide?

Answer 62

The likelihood ratio tests table provides information on the deterioration in the fit from the final model fitted in this stage to reduced models in which particular terms are removed from the model. (as we go down the list)

Question 63

what is model fitting table about?

Answer 63

the difference between the intercept and our model with x predictor and x df

Question 64

what does the chi 2 in the model fitting table mean?

Answer 64

From SPSS output “ the chi 2 statistic is the difference in -2 log likelihoods between the final model and a reduced model. The reduced model is formed by omitting an effect from the final model. The null hypothesis is that all parameters of the effect are 0” gives the difference between the model and intercept full model using the -2 log likelihood and compares it against a chi2 dist

Question 65

Log Reg uses measures of xxxxx rather than sums of squares

Answer 65

deviance

Question 66

the focus is on the

Answer 66

lack of fit

Question 67

what is -2 log likelihood ?

Answer 67

Deviance measures contrast log likelihoods using log likelihood ratios. The log likelihood is a function of the probabilities of the observed and model- predicted outcomes for each case, summed over all cases so roughly, the maximum log likelihoods of the following get compared to each other: 1. model with no predictors (only an intercept), with 2. perfectly fitting model (aka saturated model) 1. model with a set of k predictors, with 2. perfectly fitting model 1) Compute a log likelihood (LLS) value for a smaller model (one with k parameters) 2) Compute a LLB value for a bigger model (one with k+m parameters) Likelihood ratio test (LRT) statistic, LRT = -2*LLS – (-2*LLB) = -2*log(LS/LB) If the smaller model is true then the LRT statistic is distributed as χ2 with m df

Question 68

why is pseudo r2 unreliable?

Answer 68

can’t compare models (one with another) sensitive to N

Question 69

what does the likelihood ratio tests table provide?

Answer 69

Individual predictor contribution The likelihood ratio tests table provides information on the deterioration in the fit from the final model [fitted in this stage] to reduced models in which particular terms are removed from the model. So if one variable exhibits a significant value it means the model has significantly worse fit without that variable

Question 70

after finding non-significant predictors from the outcome of the likelihood ratio tests, we would?

Answer 70

re run without those predictors as they do not contribute to the model well

Question 71

so if the -2LL value gets higher, what does this indicate?.

Answer 71

indicates a worse fitting model

Question 72

what does the chi 2 column indicate in the likelihood ratio tests table?

Answer 72

The difference in -2LL values is shown in the chi-square column of the table and this is the likelihood ratio test statistic for the deterioration in model fit and it has x df which is the number of parameters associated with the main effect

Question 73

what does the Parameter estimates table give us…

Answer 73

the effect of each component in a model with all the components present - their effect, independent of the other predictors present in the model the parameter estimates table provides a test of the effect within a model containing the other terms - their independent contribution to the model over and above all the other factors in the model (like partial regression coefficients)

Question 74

so for categorical predictors in the Parameter estimates table we get …..

Answer 74

one compared to the other (which is why one is set to =0 and no values are put in)

Question 75

and continuous predictors (like age or a likert scale or whatever) are presnted….

Answer 75

on their own ….

Question 76

and the odds ratio EXP(B) reflect for continuous means ….

Answer 76

change in odds ratio for every unit increase in the variables ON THE dv

Question 77

but the Exp(B) for categorical means….

Answer 77

odds change in reference to the other category on DV

Question 78

odds ratio of 1 means…

Answer 78

no effect

Question 79

further away from 1 =

Answer 79

the more is going on

Question 80

Exp(B) values below 1 which are significant in the categorical variables mean what?

Answer 80

that the unstated reference category is more likely (women here was the reference category) “In both cases the odds of identification are lower for men than for women. The odds ratios for men:women, are given in the gendwitn=1 rows, and are 0.644 and 0.471 for identifying a suspect and identifying a volunteer respectively. These odds ratios are both significantly below 1. This pattern is consistent with a bias towards making an identification (whether accurate or not) in women compared to men.

Question 81

Exp(B) values below 1 which are significant in the continuous variables mean what?

Answer 81

The odds of xxx [sig or not] decreases for each unit increase in xxx

Question 82

if a continuous Exp(B) value was above 1 at say… 1.899, what would the interpretation be?

Answer 82

For each unit increase in xxxxx, the odds of a xxxx almost doubles

Question 83

so above main effects you could look at more parameter models, a saturated model would be all main effects and interactions, but inbetween that, you could extend a model to include …..

Answer 83

interaction terms between some of the main effects (e.g. interaction between gender and marital status)

Question 84

if we included that interaction with all the main effects, what would this model be called …

Answer 84

full factorial model (it’s full with respect to factors i.e. it has all factors, main effects and interactions, and it has all the main effects of the covariates) same as ancova -full factorial- doesnt have interactions between categorical and covariate variables only factors interact with other factors not with covariates He calls it COMPLETE model saturated would be treating every possible cell by cell combination (so age would be broken down into every single age)

Question 85

again, the difference between -2LL values for two models is a

Answer 85

likelihood ratio test statistic

Question 86

and this is distributed approximately as the

Answer 86

chi-squared distribution with degrees of freedom (df) equal to the difference in number of parameters between the two models.

Question 87

if the statistic is highly significant ?

Answer 87

= statistically significant deterioration in fit from the final model to the intercept only model.

Question 88

state why the DF - e.g. • A good answer might also explain why there are x df. The final model contains terms for breed (3 levels); the food factor (2 levels); and the breed*food interaction .

Answer 88

• A good answer might also explain why there are 5 df. The final model contains terms for breed (3 levels), which requires 2 parameters (3‑1); the food factor (2 levels) requires 1 parameters (2-1); and the breed*food interaction also requires 2 parameters. This explains the 5 df (=2+1+2).

Question 89

• This shows that the model fitted for study 1 is a full factorial model (it has all the possible factors and their interactions included). It is also a saturated model as when only factorial IVs are included (ie no covariates), and the model is full… there are ….

Answer 89

there are no more degrees of freedom.

Question 90

• the goodness-of-fit (GOF) table compares

Answer 90

the deterioration of fit between a saturated model (i.e., a model with 0 df, that provides the best possible fit to the data) and the final model for that stage of the analysis.

Question 91

The two statistics (Pearson and Deviance) calculate the goodness of fit statistic in

Answer 91

slightly different ways

Question 92

(Deviance is equivalent ……

Answer 92

to a log likelihood ratio test). The zero df for the GOF statistics indicates that the model tested is a saturated one (as it contains the same number of parameters as the saturated model with which it is compared).

Question 93

McFaddon’s ρ2 treats ….

Answer 93

Dnull - Dk as equivalent to SSregression and Dnull as equivalent to SSresidual, and thereby calculates R2 just as one would in OLS regression.

Question 94

Cox & Snell is a variation on this which reaches a maximum of

Answer 94

.75 when there is an equal N in each category of the DV,

Question 95

Nagelkeke Index divides

Answer 95

Cox and Snell’s R2 by its maximum in order to achieve a measure that ranges from 0 to 1.