S6) Reviews of Evidence

Question 1

Literature reviews of studies are used in evidence-based healthcare.

Describe the difference between these studies

Answer 1

• Narrative reviews: implicit assumptions, opaque methodology, not reproducible → biased, subjective
• Systematic reviews: explicit assumptions, transparent methodology, reproducible → unbiased, objective

Question 2

Systematic reviews involve a clearly focused question with explicit statements about which factors?

Answer 2

• Types of study
• Types of participants
• Types of interventions
• Types of outcome measures

Question 3

What is a systematic review?

Answer 3

A systematic review is an overview of primary studies that used explicit and reproducible methods

Question 4

Identify the four steps involved in a systematic review

Answer 4

⇒ Systematic literature search

⇒ Selection of the materials

⇒ Appraisal

⇒ Synthesis

Question 5

What is a meta-analysis?

Answer 5

A meta-analysis is a quantitative synthesis of the results of two or more primary studies that addressed the same hypothesis in the same way

Question 6

Illustrate the relationship between a systematic review and a meta-analysis

Answer 6

A systematic review will not necessarily include a meta-analysis if, for example, clinical heterogeneity is too great

Question 7

State the four purposes of a meta-analysis

Answer 7

• To facilitate the synthesis of a large number of study results
• To systematically collate study results
• To reduce problems of interpretation due to variations in sampling
• To quantify effect sizes and their uncertainty as a pooled estimate

Question 8

To ensure quality criteria, a meta-analysis should have a formal protocol.

What should be specified?

Answer 8

• Compilation of complete set of studies
• Identification of common variable or category definition
• Standardised data extraction
• Analysis allowing for sources of variation

Question 9

How does one interpret a forest plot?

Answer 9

• Individual odds ratios (squares) with their 95% CI (lines) are displayed for each study
• Size of square is in proportion to the weight given to the study
• The (diamond) is the pooled estimate with the centre indicating the pooled odds ratio (dotted line) and the width representing the pooled 95% CI
• The (solid line) is the null hypothesis OR

Question 10

What are the three major problems with a meta-analysis?

Answer 10

• Heterogeneity between studies
• Variable quality of the studies
• Publication bias in selection of studie

Question 11

There are two approaches to calculating the pooled estimate odds ratio (OR) and its 95% CI.

Describe both the fixed effect model and the random effects model

Answer 11

• Fixed effect model – assumes that the studies are estimating exactly the same true effect size
• Random effects model – assumes that the studies are estimating similar, but not the same, true effect size

Question 12

Compare and contrast the fixed effect model with the random effects model in light of the following:

• Point estimate
• 95% Confidence Interval
• Weighting of the studies
• Hypothesis test for heterogeneity

Answer 12

• Point estimate e.g. odds ratio – is often similar in both models
• 95% CI – is often wider in the Random Effects Model
• Weighting of the studies – is more equal between the studies in the Random Effects Model i.e. greater weighting towards small studies
• Hypothesis test for heterogeneity – low statistical power to detect heterogeneity, often use 10% significance level

Question 13

What are the causes for variable quality?

Answer 13

• Poor study design
• Poor design protocol
• Poor protocol implementation

Question 14

What is the reason for publication bias?

Answer 14

Studies with statistically significant / ‘favourable’ results are more likely to be published than those studies with non-statistically significant / ‘unfavourable’ results

Question 15

What are the consequences of publication bias?

Answer 15

• Any systematic review / meta-analysis can be flawed by such bias
• Publication bias leads to a biased selection of studies towards demonstration of effect

Question 16

Identify the three steps in the method of identification of publication bias in the selection of studies

Answer 16

⇒ Check meta-analysis protocol for method of identification of studies

⇒ Plot results of identified studies against a measure of their size i.e. a funnel plot

⇒ Use a statistical test for publication bias

Question 17

How might one interpret a funnel plot for publication bias?

Answer 17

• If no publication bias, then the plot will be a ‘balanced’/symmetrical funnel
• Smaller studies can be expected to vary further from the ‘central’ effect size
• Publication bias is likely to exist if there are few small studies with results indicating small or ‘negative’ measure of effect