META-ANALYSIS

Question 1

Essentially, how do meta-analyses work?

Answer 1

• We compute effect size and variance for each study

- We take the weighted mean of these effect sizes, usually by giving more weight to precise studies (1/total variance)

Question 2

What would happen if we didn’t weigh in meta-analyses?

Answer 2

Simpson’s paradox! Which is when we take into account a lurking explanatory variable, the effect changes direction.

Question 3

What is the difference essentially between fixed effects and random effects in terms of variance calculation?

Answer 3

Fixed: we calculate the variance based only on the included studies
Random: we assume that the studies included are only a random sample of all possible studies

Question 4

What is heterogeneity testing?

Answer 4

Assessing the consistency of effects across studies

Ho: all studies are evaluating the same effect

Question 5

What is Cochran’s Q Statistic?

Answer 5

Traditional statistic to test for heterogeneity

It sums (meta estimate - study estimate)^2, weighting each study in the same way as for the pooled estimate

Question 6

What is the problem with Cochran’s Q statistic?

Answer 6

Problems of power: It’s has either low power (small N) or is too sensitive (larger N)

But there’s no point in testing for heterogeneity - what matters is its impact on conclusions

Question 7

What is I2 statistic?

Answer 7

The % observed total variation across studies due to heterogeneity rather than chance

Q-df/Q x 100%

Question 8

What are the advantages of the I2 statistic?

Answer 8

• It does not depend on the number of studies
• It can be compared between meta-analyses regardless of the N and the operationalization of the outcome and effect measure
• Can be accompanied by uncertainty interval
• Simple to calculate and may be derived from published meta-analyses

Question 9

What does a fixed effect meta-analysis calculate?

Answer 9

The best estimate of assumed common treatment effect

Question 10

What does a random effect meta-analysis calculate?

Answer 10

The average from the distribution of treatment effects across studies, with a prediction interval

Question 11

What is the nature of the prediction interval of a random effect meta-analysis?

Answer 11

The predicted range for true treatment effect in individual study:
u +/- t(k-2)*sq(T^2 + SE(u)^2)

• u: summary estimate
• k: number of studies included
• T: estimate of between study standard deviation

Question 12

Why use a t-distribution in the prediction interval for a random effect meta-analysis?

Answer 12

It accounts for the uncertainty of T, the estimate of bw study standard deviation

Question 13

In summary, the prediction interval accounts for:

Answer 13

1. The uncertainty of the summary estimate
2. The estimate of bw study SD in true treatment effect (T)
3. The uncertainty in the bw study SD estimate (T) itself

Question 14

4 biases in meta-analyses?

Answer 14

1. Publication bias
2. Time lag bias
3. Language bias
4. Citation bias

Where positive results are more likely to be published, quickly, in English, and cited.

Question 15

What does a funnel plot look like and how do we interpret it?

Answer 15

x axis: Log risk ratio (effect size)
y axis: Standard error (study precision)

if we have a gap in negative results from small (unprecise) studies (bottom left), we have a publication bias problem.

Question 16

Fixed effects

Answer 16

• Assumes all studies are identical repeats of same experiment
• Only accounts for within study error

Question 17

Random effects

Answer 17

• Assumes studies come from an underlying distribution

- Accounts for within and between study variance