Intro to multi-level models (MLM)

Question 1

Clustering

What is clustered data?

Answer 1

= when our observations have some sort of grouping that is NOT something we are interested in studying

This is important as the observations in one cluster are likely to be more similar to each other than to observations in different clusters

For example:
- children within schools
- observations within individuals

This can quickly become complicated with the more levels you have

Question 2

Clustering

What is the intra-class correlation coefficient (ICC)?

Answer 2

It is a ratio of:
variance between groups : total variance

ICC can take values between 0 and 1. The larger the ICC, the lower the variability is within the clusters compared to the variability between clusters

For example:
ICC of 0.48 means that 48% of our variance is attributable to by-cluster differences

Question 3

Clustering

Why and how is clustering a problem?

Answer 3

WHY:
It is something systematic that our model should (arguably) take into account

HOW:
standard errors are often smaller which also means:
- CIs will be too narrow
- t-statistics will be too large
- p-values will be too small

Question 4

Clustering

What is wide data?

Answer 4

observations are written across separate columns (not the preferred method)

Question 5

Clustering

What is long data?

Answer 5

there is one column for all observations and the different columns tell us which cluster an observation belongs to

Question 6

Clustering

Dealing with Clustering
What is complete pooling?

Answer 6

Ignoring clustering
= information from all clusters is pooled together to estimate x

• basically, takes everyone’s data, pools it together and then plots it as one line on a graph

This is not the best method as clusters can show different patterns which this method does not account for
- also residuals are NOT independent

Question 7

Clustering

Dealing with Clustering:
What is no pooling?

Answer 7

Fixed effects models
= information from a cluster contributes to an estimate for that cluster (and ONLY that cluster)

• information is not pooled

This method is good as it has lots of estimates
BUT it is flawed as no pooling would include results that might be anomalous
also it has less statistical power due to having more parameters

Question 8

Clustering

Dealing with Clustering:
What is partial pooling?

Answer 8

Random Effects models
= cluster level variance in intercepts and slope is modelled as randomly distributed around fixed parameters.
- Effects are free to vary by cluster but information from all clusters contributes to an overall fixed parameter

Rather than estimating differences for each cluster, we are estimating the variation (or spread of distributions) of intercept points

Question 9

MLM

What is multilevel regression?

Answer 9

used for observation j in group i

used for data structures when we’ve observed things that happen at multiple levels
e.g. children in classes in schools etc

Question 10

MLM

Multiple regression equation

Answer 10

It looks similar to simple regression but we now need a 2 level equation

Level 1:
yij = β0i + β1ixij + ε

Level 2:
β0i = γ00 + ζ0i
β1i = γ10 + ζ1i

Where:
- γ00 is the population intercept and ζ0i is the deviation of group i from γ00
- γ10 is the population slope and ζ1i is the deviation of group i from γ10

Basically
γ means there is a fixed number for the whole population
ζ accounts for the deviation of each individual group (random effects)

Question 11

MLM

Assumptions of multiple regression:

Answer 11

we now assume ζ0, ζ1 and ε to be normally distributed with a mean of 0

Question 12

MLM

What are fixed effects?

Answer 12

Items that do not vary by cluster are fixed effects
e.g. γ00 or γ10

If we repeated the experiment we would use the same levels
Desired inference:
the conclusions refer to the levels used

Question 13

MLM

What are random effects?

Answer 13

common definition = “we allow (?) to vary by (?)”

ζ is considered random as it is considered a random sample from a larger population

If we repeated the experiment different levels would be used
desired inference:
the conclusions refer to a population from which the levels used are just a (random) sample

This whole thing is random effects in R:
…. ( random intercept + random slope | grouping structure)

Question 14

MLM

Radom intercept vs Random slope (in R)

Answer 14

Random intercept:
lmer(y ~ 1 + x + (1|g), data = df)

Random intercept and slope:
lmer(y ~ 1 + x + (1 + x |g), data = df)

Question 15

MLM

Advantages of MLM

Answer 15

MLM can be used to answer mutli-level questions, for example:

1. Do phenomena at level X predict outcomes at level Y?
   e.g “does left vs right handedness predict variation in reaction times?”
2. Do phenomena at level X influence effects at level Y?
   e.g. “does being mono vs bilingual influence grades over the duration of schooling?”
3. Do random variances covary?
   e.g. “do people who have higher cognitive scores at the start of the study show less decline over the duration of the study than those who started with lower scores?”

Question 16

MLM

lmer output:

Answer 16

Fixed effects = one fixed model line

Random effects = random (individual) deviations around the fixed line (assumed to be normal)

Residual = captures the final step from individual line to individual observations

Question 17

MLM

ICC in lmer

Answer 17

Obtained by fitting an intercept only model in lmer as ICC is conditional on random intercepts (so the inclusion of random slopes would influence )

From the random effects in your model summary:
ICC = intercept / ( intercept + slope or residual value )

Question 18

MLM

what is marginal R squared?

Answer 18

= variance explained due to fixed effects

Question 19

MLM

what is conditional R squared?

Answer 19

= variance explained due to fixed and random effects

Question 20

MLM

Model estimation

Answer 20

MLM are too complicated to calculate using a closed form solution so instead we estimate all the parameters using an iterative procedure of maximum likelihood

Question 21

MLM

Maximum Likelihood estimation (MLE)

Answer 21

Aim = find the values for the unknown parameters that maximise the probability of obtaining the observed data

How = done by finding values that maximise the log-likelihood function

treats fixed effects as KNOWN when estimating the variance components at each iteration
- this can lead to biased estimates of variance components

Question 22

MLM

Restricted maximum likelihood estimation (REML)

Answer 22

estimates the variance components first and then separates the estimation of fixed and random effects

• this leads to less bias estimates of the variance components
  better for small sample sizes

Question 23

MLM

What are convergence warnings?

Answer 23

They come into play when model when the optimiser we are using either can’t find a suitable maximum, or gets stuck in a singularity (think of it like a black hole of likelihood, which signifies that there is not enough variation in our data to construct such a complex model)

Question 24

MLM Inference

p-values in lmer?

Answer 24

In simple lm we test the reduction in SSresidual which follows an F-distribution with known df.

only in very specific conditions in lmer will we have known df.
Parameter estimate in MLM are MLE/REML estimates which means it is:
- unclear how to calculate denominator degrees of freedom (DDF)
- also unclear as to whether t-statistics would even follow an f-distribution

We need other options for inference

Question 25

MLM Inference

Options for inference:

Approximating DDF
- Kenward-rogers

Answer 25

Kenward rogers:
- models must be fitted with REML
- adjusts SEs to avoid small sample bias
- approximated denominator df (may not be whole number)

Question 26

MLM Inference

Options for inference:

Likelihood based methods
- profile likelihood confidence interval

Answer 26

Models need to be fitted with MLE

Evaluates the curvature of the likelihood surface at the estimate
- sharp curve = more certainty in estimate
- gradual curve = less certainty

Question 27

MLM Inference

Options for inference:

Likelihood based methods
- likelihood ratio tests

Answer 27

Models need to be fitted with MLE
- not good for small sample sizes

Uses anova()
Compares loglikelihood of two competing models
- ratio of two likelihoods is asymptotically (as n increases towards infinity) chi squared distributed

Question 28

MLM Inference

Options for inference:

Bootsrap

Answer 28

Parametric bootstrap:
- confidence interval
- likelihood ratio test

Case based bootstrap
- confidence interval

Question 29

MLM inference

MLE vs REML

Answer 29

Fit with ML if:
- models differ in fixed effects only
- models differ in BOTH fixed and random effects
- you want to use anova()

Fit with REML is:
- models differ in random effects only

Question 30

Assumptions and Diagnostics

Assumptions in lm

Answer 30

The general assumptions in lm are
“mean of 0 and constant variance”

Remember: LINE?

Question 31

Assumptions and Diagnostics

Assumptions in MLM

Answer 31

Similar to lm - the general idea is the same of:
“error is random”

but now we have residuals at multiple levels!
- we have our overall (fixed effects) line
- then random effects lines of how much the group line is different from the overall line slope
- then around the random effects lines we have the residuals of individual points

Question 32

Assumptions and Diagnostics

Assumption plots:
plotting residual vs fitted values

Answer 32

plot ( model , type = c ( “p” , “smooth” ) )

Question 33

Assumptions and Diagnostics

Assumption plots:
QQplots

Answer 33

Used to check normality
- we want the dots to follow the line

qqnorm(resid(model))
qqline(resid(model))

Question 34

Assumptions and Diagnostics

Assumption plots:
Scale-location plots

Answer 34

Measure of spread
- we just want the line to be horizontal (it doesn’t matter where it sits)

plot(model,
form = sqrt(abs(resid(.))) ~ fitted(.),
type = c(“p”,”smooth”))

Question 35

Assumptions and Diagnostics

Assumption plots:
Plotting by cluster

Answer 35

Used just to look for systematic patters

Question 36

Assumptions and Diagnostics

Assumption plots:
Quick assumption check

Answer 36

performance :: check_model(model)

this gives us an overview but isn’t used in formal write-ups

Question 37

Assumptions and Diagnostics

Troubleshooting:
Model mis-specification?

Answer 37

if assumptions look violated, check the model is correct
e.g.
- are the interaction terms needed
- does that variable vary by cluster

Question 38

Assumptions and Diagnostics

Troubleshooting:
Transformations?

Answer 38

(not massively recommended)

Transforming your outcome variable may help satisfy model assumptions
- but this may come at the expense of interpretability

there are many methods
e.g. BoxCox = finds the ‘best’ transformation
- after this we can only refer to y as BoxCox transformed y and we have no way of knowing if our transformed y is meaningful

Question 39

Assumptions and Diagnostics

Troubleshooting:
Bootstrap?

Answer 39

Same basic principles as in lm

if we are concerned our errors are non-normal or heteroskedastic and we have a LARGE sample size, it might be a good option
BUT if there are effects with mis-specification (e.g. the effect is non-linear) bootstrapping won’t help

Question 40

Assumptions and Diagnostics

Troubleshooting:
Types of Bootstrapping
Parametric bootstrap

Answer 40

= resample based on the estimated distribution of parameters

assumes explanatory variables are fixed and that the model specification and distributions are correct
- not very helpful in assumption violations

Question 41

Assumptions and Diagnostics

Troubleshooting:
Types of Bootstrapping
Resample Residuals

Answer 41

y* = y hat + ε hat
- sampled with replacement

assumes explanatory variables are fixed and that the model specification and distributions are correct
- not very helpful in assumption violations

Question 42

Assumptions and Diagnostics

Troubleshooting:
Types of Bootstrapping
Case based bootstrapping

Answer 42

= resample cases
- minimal assumptions other than that we have specified the hierarchical structure of our data

BUT this presents us with the issue of do we sample individual observations? do we sample clusters? or both?

For example, in R to bootstrap participants (clusters) but not their observations we include

resample = c (TRUE,FALSE)

Question 43

Assumptions and Diagnostics

Influence
What are influential cases?

Answer 43

high leverage cases = are able to direct our model line in a certain way

high outlier = points that fall far from our model line and other observed data points

high influence = high leverage + high outlier
- the case is far from our other observed data points and pulls the model line in a direction that misrepresents the rest of the observed data
- cook’s distance

Both observations (level 1) and clusters (level 2) can be influential

Question 44

Assumptions and Diagnostics

Influence
Level 1: Influential points

Answer 44

plot QQplot of model

Diagnostics package
library(HMLdiag)
infl1 <- hml_influence(model, level = 1)

• dot plot (of cook’s distance) = points beyond the red line can be considered influential
  dotplot_diag(infl1$cooksd, cutoff = “internal”)

Question 45

Assumptions and Diagnostics

Influence
Level 2: Influential clusters

Answer 45

If we have multiple observations for each participant, each participant can be considered a cluster
So to determine an influential cluster:

infl2 <- hml_influence(model, level = “ppt”)
dotplot_diag(infl2$cooksd, cutoff = “internal”, index = infl2$ppt)

This will provide a graph of clusters (participants) scaled on influence

Question 46

Assumptions and Diagnostics

Influence
Sensitivity analysis

Answer 46

Would our whole conclusion change if we excluded an influential case?

We make a model with and without the influential case and compare them
- if your conclusion does not change, don’t mention it
- if your conclusion changes, mention it in the discussion and look closer at the influential case to determine why it is so influential

Question 47

Centring Predictors in MLM

What is centring?

Answer 47

We can recentre our data so that any value forms the new 0 point

common versions of this are:
- mean centring
- centring so our data starts at 0 (so our model is not trying to predict the 0 point)

Question 48

Centring Predictors in MLM

What is scaling?

Answer 48

Suppose we have a variable for which the mean is 0 and the sd id 15 - we can change the scale of our data so 1 unit change in x is equivalent to 1 unit change in sd

Question 49

Centring Predictors in MLM

Group mean centring

Answer 49

in MLM we have multiple means to work with:

Grand mean = mean of all observations (regardless of cluster)

Group means = mean of each cluster

Group mean centring = take each individual observation in a group and minus the group mean

Question 50

Centring Predictors in MLM

Within effects

Answer 50

When looking at the graph, you are looking at the differences between individual observations that contribute to one line

For example, in a study of how anxiety effects drinking (for clusters) of participants
“is being more nervous (than you usually are) associated with higher alcohol comsumption”

To answer this, we would group mean centre anxiety to plot the clusters (participants) against their average anxiety levels

Question 51

Centring Predictors in MLM

Between effects

Answer 51

When looking at the graph, you look at the differences between the cluster lines

For example, in a study of how anxiety effects drinking (for clusters) of participants
“is being generally more nervous (than other participants) associated with higher alcohol consumption

To answer this, we plot group means and then compare them

Question 52

Centring Predictors in MLM

When do we need to consider within and between effects?

Answer 52

• when we have a predictor that varies within a cluster
• when we have different average levels of x (typically occurs in observations rather than experiments)
• when our RSQ concerns x

Question 53

Random Effect Structures

What is a nested structure?

Answer 53

Things in that structure belong only to that structure

e.g. children will belong to one class, in one school, in one district etc

Question 54

Random Effect Structures

Nested random effects structures:

Answer 54

Imagine the children in classes in schools example
We can write their random effects structures in R a few ways:

(1| school ) + ( 1 | class )
( 1 | school ) + ( 1 | class : school)

( 1 | school / class ) = “group by school and within that group by class”
- This can only be used if group labels are unique
- for example, if the children in each class are labelled 1-30 we wouldn’t know who was who if we used this method

Question 55

Random Effect Structures

What is a crossed structure?

Answer 55

basically the opposite of nested structures
the things in one cluster can also be in other clusters

e.g. we have multiple participants complete 5 tasks multiple times
- observations here can be clustered by participant and by task

Question 56

Random Effect Structures

Crossed random effects structures:

Answer 56

Imagine the multiple participants complete 5 tasks multiple times example

We write the crossed random effects structure in R as follows:
… + ( 1 | ppt ) + ( 1 | task )

Question 57

Model Building

Maximal structures
What is a maximal model?

Answer 57

maximal model = the most complex structure that you can fit to the data

• everything that can be modelled as a random effect is done so
• everything in the fixed effects is ‘up for grabs’ for the random effects
• requires sufficient variance (which if often not the case)

in R:
we use isSingular(maxmodel) to see if the model has converged to a sensible answer
- if the output is TRUE there is an issue and we need to simplify our maximal model

Question 58

Model Building

Maximal structures
What do you do if a model won’t converge?

Answer 58

Don’t report results from a model that won’t converge! you can’t trust its estimates

Instead:
- check the model specification = do your random effects make sense?
- try a different optimizer
- adjust the max iterations
- stop the tolerances
In most cases our model is too complex and we just have to simplify it

Question 59

Model Building

Maximal structures
What is an optimizer?

Answer 59

= the method by which the computer finds the best fitting model for our MLM

we can try all optimizers at once in R using:
summary(allFit(model))
this may help us choose an optimizer that allows our model to converge

throughout DAPR3 they’ve used ‘bobyqa’ like literally all the time

Question 60

Model Building

Maximal structures
Deciding on random effect structures
Selection based

Answer 60

Use a criterion for model selection (e.g. LRT, AIC, BIC etc.) to choose a random effect structure that is supported by the data
- these are parsimony corrected so have more power

Question 61

Model Building

Maximal structures
Deciding on random effect structures
Keep it maximal

Answer 61

start with the maximal model

remove random effects with the least variance until the model converges

this means we’re trying to fit the most complicated model that we can given the variance in our data (risks overfitting)

Question 62

Model Building

Maximal strucutres
Simplification

Answer 62

• extract the random effects
  in R = VarCorr(maxmodel)
  Look for:
• small variances / sd
• correlations of 1 or -1
• Consider removing the more complex random effects first (e.g. interaction term)
• categorical predictors with 2+ levels are ‘more complex’ (as they require more parameters)
• remove higher level random effects if needed (if you have multiple levels of nesting, you’ll have fewer groups as you go up the levels)
• subjective choice = which simplification can you most easily accept

Question 63

Model Building

Random effects correlation

Answer 63

Removing random effects correlations simplifies the model

Correlations between our random effects can alter our model results

we can remove the correlations but still see the random effects by using || in our model instead of | for example:
… + ( 1 + x || y )

Question 64

RSQs with MLM

Model specification
LMER ( outcome ~ fixed effects + ( random effects | grouping structure ) , data = data)

Answer 64

LMER = how is it measured

before fitting a model think: do we need to centre or scale

Question 65

RSQs with MLM

Model specification
lmer ( OUTCOME ~ FIXED EFFECTS + ( random effects | grouping structure ) , data = data)

Answer 65

OUTCOME
what are we interested in explaining and predicting

FIXED EFFECTS
- what variables are we interested in explaining by this
- are our questions about the effect if our predictors specifically in reference to group means of predictors? etc.

Question 66

RSQs with MLM

Model specification
lmer ( outcome ~ fixed effects + ( RANDOM EFFECTS | grouping structure ) , data = data)

Answer 66

RANDOM EFFECTS
which of our fixed effects can vary for our random groups?

• does a single group have multiple distinct values of x?
• what can we imagine a slope for?

Question 67

RSQs with MLM

Model specification
lmer ( outcome ~ fixed effects + ( random effects | GROUPING STRUCTURE ) , data = data)

Answer 67

GROUPING STRUCTURE
in what different ways can we group our data?

• of the ways we can group our data, which are of specific inferential interest?
• of the ways we can group our data, which groupings do we think of as a random sample of a general population?
  - are these groupings nested?
  - are the labels unique? etc.

Question 68

RSQs with MLM

Model fitting

Answer 68

model issues = check for convergence and singular fit (adjust accordingly)

model assumptions
- can use check_model to check everything looks right
- normality of residuals = QQplot
- influence = cook’s distance / dot plot

plots are preferable as statistical tests can be overly sensitive

Question 69

RSQs with MLM

Interpretation and Inference
Fixed effects

Answer 69

use: fixef(model) to get fixed effect values

Interpretation = using plot_model to plot your fixed effect may help you make sense of the fixef(model) output

Inference
Tests
- model comparison
- parameter estimates
Methods
- df approximations (e.g kenward-rogers)
- Likelihood ratio tests
- Bootstrap

We can use any of these to make inferences about our fixed effects

Question 70

RSQs with MLM

Interpretations and Inference
Random effects

Answer 70

we use random effects to add context to our results
For example, adding random intercepts and slopes to a graph can help provide context of the actual trends in the data - helping us answer our RSQ

Question 71

RSQs with MLM

Reporting
The Analytical Process

Answer 71

1) data cleaning/removal/transformations are performed prior to analysis

2) unplanned transformations or removals performed in order to meet assumptions

3) specify all fixed effects (explanatory variables and covariates) linking to the RSQ/hypothesis

4) plan a structure of random effects to be fitted and the procedure used to decide final random effects structure if the model does not converge

5) state clearly relevant test / comparison and link this to RSQ / hypothesis

Question 72

RSQs with MLM

Reporting
Results

Answer 72

1) software packages used to fit models
2) estimation method
3) optimizer used

4) if the proposed model failed to converge, the steps used to reach final model

For final model
5) all parameter estimates for fixed effects (e.g. coefficients, SEs, CIs, t-stats/df/p-values if used)

Random effects
6) variance/sd for each random effect, residual variance, correlations/covariates if modelled

Question 73

RSQs with MLM

Reporting
tests and tables

Answer 73

Tables help a lot but results given in the table must be included within the written interpretations to provide the reader with the context of each number’s meaning