Lecture 6: Meta-analysis, meta-regression and limitations of meta-analyses

Question 1

What is a meta-analysis?

Answer 1

A procedure to combine results of various studies on the same topic, provides a reliable overview of literature. Can give insight into which variables explain the differences in effect sizes between studies (moderators), important for clinical practice as there is no time to read a lot and less access to literature. Very popular and has lots of influence

Question 2

What are the disadvantages of a narrative review?

Answer 2

• focus on p-values in original studies
• unable to deal with inconsistent results due to reviewer bias
• does not take into account the reliability of studies
• they only used published articles which have publication bias

Question 3

What are the advantages of a meta-analysis?

Answer 3

• p values are irrelevant so there is a focus on effect sizes
• all the results are include so inconsistent results just reduce the overall effect
• studies are weighted with reliability
• literature is extensively searched, can use unpublished studies but not always and tests for publication bias

Question 4

What is pooling?

Answer 4

Calculating a mean effect

Question 5

What are the units of analysis used in meta-analysis?

Answer 5

Looks at the effect size of studies (to overcome the issue of not using the same test or scale). The effect size can estimate the true effect size in a population

Question 6

How is more weight given in a meta-analysis?

Answer 6

More reliable studies are given more weight, so they also appear larger

Question 7

What is y(k)?

Answer 7

The observed effect size of study k

Question 8

What is b0?

Answer 8

The (un)weighted mean effect size so the estimated mean of the true effect in the population

Question 9

How does reliability affect the 95% CI?

Answer 9

95% CI is based on SE, which is highly dependent on sample size. The larger N so the more precise estimate of the true effect will result in a smaller SE and smaller CI which results in more weight.

Question 10

What is the effect size of each study?

Answer 10

The estimate of the true effect size in the population and studies differ in the precision/reliability of that estimation

Question 11

What is the fixed-effect model?

Answer 11

Argues that studies are from 1 single homogenous population, so there is only 1 true effect size. Observed effect sizes are the estimators of the same true effect size. Any differences in the observed effect sizes are due to sampling error. So there is only 1 error term

Question 12

What is the random-effects model?

Answer 12

Argues that studies are form a universe of population. Each sub-population has a different true effect sizes, so there is a distribution of true effect sizes. Studies are random samples from that distribution. There are two error terms: sampling error (spread within a study) and variation in true effect sizes (spread between studies

Question 13

How is study weight calculated in a fixed-effect model?

Answer 13

SE is the sampling error and variance is SE^2. Weight for k = 1/ variance (within study)

Question 14

How does reliability affect weight in a fixed-effect?

Answer 14

SE indicates precision (reliability) of estimation of the true effect size for study k, which is strongly determined by N. The smaller N, the greater the within-study variance. The greater within-study variance, the smaller the weight

Question 15

How can you calculate the weight for a random-effects model?

Answer 15

Weight study k= 1/ (variance within+ variance between)

Question 16

What is the variance between studies?

Answer 16

It is one value which indicates variation of true effect sizes between studies. The more the observed effect sizes differ, the greater the between-studies variance. The greater the between-studies variance, the smaller weight

Question 17

How is weight different in the random-effects model?

Answer 17

Small studies have relatively more weight and larger studies have relatively less weight, as the between studies variance also plays a role in the study

Question 18

What happens when there is no between study variation?

Answer 18

The fixed effect meta-analysis is the same as the random-effects meta-analysis

Question 19

How to test whether the summary effect is significant?

Answer 19

Calculate SE for the fixed-effects model = sqrt (average variance within study)
SE for random effects model= sqrt (average variance within study+ variance between studies)
Which can be used to calculate the Z value= M/se

Question 20

What is the confidence interval?

Answer 20

It indicates precision/reliability. With more participants, the average within study variance is lower while the SEm and CIm smaller. More heterogeneity will result in a larger variance between studies and a larger SEm and CIm. So the CIm will always be larger in random effects than the fixed effects model

Question 21

What is important to pay attention to?

Answer 21

The size of the mean effect, the significance of the mean effect and CI range

Question 22

How can we explain the differences in effect sizes?

Answer 22

Study characteristics so moderators can explain why the effect of a treatment in larger than in another study

Question 23

What is the interpretation of y(k)?

Answer 23

effect size (Hedges’ g) for difference in mean PTSD symptoms at
posttest between IRT and WLC in study k

Question 24

What is the interpretation of x(k)?

Answer 24

the average study characteristic values in study k

Question 25

What is the interpretation of the bs?

Answer 25

b0: expected effect size (g) when study characteristic = 0
b1: difference in expected effect size (g) if study characteristic increases by 1

Question 26

What is the interpretation of the error?

Answer 26

difference between observed and expected effect size in study k

Question 27

What is the meta-regression model?

Answer 27

y(k) = b0 + b1*x(k) + error(k), where x’s are study characteristics and k is a study

Question 28

What is the meta-analysis model?

Answer 28

model with intercept only
y(k) = b0 + error(k), where k is a study
b0 = mean/average effect size for all studies

Question 29

How can you detect a possible publication bias in a meta-analysis?

Answer 29

Use a funnel plot to see whether any data is missing (so if there is no funnel shape). The higher on the y axis, the more reliable the study (larger N) and the smaller SE.

Question 30

Why does this study have a smaller weight?

Answer 30

This study has tested the fewest subjects and therefore estimates the true effect size in the population with the least accuracy. The weight of a study = 1/variancewithin study. When N becomes smaller, the within-study variance increases (more sampling error) and the weight decreases.

Question 31

Why would an effect be smaller in a regression analysis compared to a meta-analysis?

Answer 31

In a simple regression analysis, the reliability of the studies isn’t taken into account. The two smallest effect sizes are belonging to the studies with the smallest sample size. Therefore, in a meta-analysis these studies receive less weight and therefore have less influence. However, in the regression analysis each study has the same weight resulting in a slightly smaller mean effect.

Question 32

Why is the estimation of a mean effect less likely to be significant in a random effects model than in a fixed-effect model?

Answer 32

The random-effects meta-analysis not only takes into account the unreliability of studies but also the spread in effect sizes between studies. Accordingly, the standard error of the mean increases, resulting in a larger confidence interval and smaller Z-value, making it less likely to be significant.

Question 33

How would you report a mean effect size?

Answer 33

In the iCBT group, participants reported on average fewer PTSD symptoms on the post-test compared to participants in a control group. This is a large effect. Moreover, the range for possible true effect sizes lied between an almost large effect (-0.71) and a large effect (-1.10), so we are quite sure that we are dealing here with a large effect.