S7 L2 Introduction to systematic reviews and meta analysis

Question 1

Why is evidence based healthcare improtant?

Answer 1

• Healthcare services and interventions should be based on best available evidence
• Best available evidence should be based on rigorously conducted research
• Primary research studies e.g. RCT
• Literature reviews of studies
  → Narrative reviews: implicit assumptions, opaque methodology, not reproducible → biased, subjective
  → Systematic reviews: explicit assumptions, transparent methodology, not reproducible → unbiased, objective
• Decision analysis
  → Harm and benefits
  → Cost-effectiveness

Question 2

What are systematic reviews?

Answer 2

Clearly focused question 
Explicit statements about:
- Types of study 
- Types of participants 
- Types of interventions
- Types of outcome measure 
Systematic literature search 
Selection of the materials
Appraisal 
Synthesis (possibly including a meta-analysis)

Question 3

What are the key aspects to a systematic reviews?

Answer 3

Credible source of evidence

Explicit, transparent and reproducible

Question 4

What is the difference between systematic review and meta-analysis?

Answer 4

Systematic review→ overview of primary studies that used explicit and reproducible methods
Meta-analysis→ A quantitative synthesis of the results of two or more primary studies that addressed the same hypothesis in the same way

Question 5

What is the purpose of meta-analysis?

Answer 5

• To facilitate the synthesis of a large number study result
• To systematically collate study results
• To reduce problems of interpretation due to variations in sampling → get a bigger sample size
• To quantify effects sizes and their uncertainty as a pooled estimate → one CI, p value, RR/OR, help determine whether a given drug is actually useful → conflicting information/sources

Question 6

How is the quality of meta-analysis determined?

Answer 6

Formal protocol specifying:

• Compilation of complete set of studies
• Identification of common variable or category definition → compare like for like
• Standardised data extraction
• Analysis allowing for source of variation

Question 7

How do you calculate a pooled odds ratio?

Answer 7

• Odds ratio and their 95% CIs are calculated for all studies in meta-analysis
• These are then combined to give a pooled estimate odds ratio using statistical computer program
• Studies are weighted according to their size and the uncertainty of their odds ratio (narrower CI→ greater weight to result)

Question 8

What does a forest plot allow?

Answer 8

Visual representation of odds ratio of each study and meta analysis

• Individual odds ratio represented as squares with their 95% CI lines displayed for each study
• Size of square is in proportion to the weight given to the study
• The diamond i the pooled estimate- centre (dotted line)= pooled odds ratio, width represents the pooled 95% CI
• Solid line us the null hypothesis OR

Question 9

What are the problems with meta-analysis?

Answer 9

• Heterogeneity between studies
  → Modelling for variation → fixed effect model vs random effect model
  → Analysing the variation → sub-group analysis
• Variable quality of the studies
• Publication bias in selection of studies

Question 10

How can heterogeneity between studies be modelled for?

Answer 10

• Fixed effect model → assumes that the studies are estimating exactly the same true effect size (differences between studies only due to random variation) → little/no heterogeneity
• Random effects model → assumes that the studies are estimating similar, but not the same, true effect size → better for greater heterogeneity between (still relatively small)
  Lot of heterogeneity - meta analysis not appropriate

Question 11

What does the fixed effect model look like?

Answer 11

True effect → calculated based on minimal difference between each of the studies (vertical solid line)
Study result (dot either side of line)
Random error → difference between the study result and true effect (horizontal line)

Question 12

What does a random effects model look like?

Answer 12

True mean effect→ mean of all studies (vertical solid line)

Each study result has its own true trial specific effect (vertical dotted line)

Question 13

What are the difference in the fixed effect and random effect data?

Answer 13

• Point estimate → often similar (not always) in both random and fixed effect model
• 95% CI → often wider in the random effects model than the fixed effect model
• Weighting of the studies → more equal between the studies in the random effects model than in the fixed effects model (greater weighting towards smaller studies)

Question 14

How can variation be analysed?

Answer 14

Random effect model- accounts for it but does not explain it
Sub group analysis can help to explain heterogeneity which may provide further insight into the effect of a treatment or exposure
→ study characteristic (e.g. year of publication, length to follow up, % female population)
→ participation profile - where the data is analysed by types of participants (e.g. subgroup of males, females, adults, children)

Question 15

What are the issue with meta analysis?

Answer 15

Variable qualities of the studies
- Due to:
→ poor study design
→ poor design protocol (way it was conducted could cause bias)
→ poor protocol implementation (didn’t follow protocol)
- Some studies are more prone to bias and confounding than others:
→ randomised controlled trials
→ non-randomised controlled trials
→ cohort studies
→ case-control studies

Question 16

How can the variability in quality of studies be accounted for?

Answer 16

Two approached used:
1- define basic quality of standard and only include studies satisfying this criteria
2- score each study for its quality and then
→ incorporate the quality score into the weighting allocated to each study during the modelling so that higher quality studies have a greater influence on the pooled estimate
→ use sub-group analyses to explore differences e.g. high quality studies vs low quality studies

Question 17

How do you assess the quality of a study?

Answer 17

For any RTC many scales are available 
Main components are:
- Allocation methods e.g. randomisation?
- Blinding and outcome assessment 
- Patient attrition e.g. intention to treat (ITT) 
- Appropriate statistical analysis

Question 18

Who assess quality?

Answer 18

> 1 assessor

handling disagreements

Question 19

What is publication bias?

Answer 19

• Studies with statistically significant or ‘favourable’ results more likely to be published than those studies with non-statistically significant or ‘unfavourable’ results- applies particularly to smaller studies
• Publication bias leads to selection bias towards studies which show an effect

Question 20

What are the methods of identification of publication bias?

Answer 20

• Check meta-analysis protocol for method of identification of studies → should include searching and identification of unpublished studies
• Plot results of identified studies against a measure of their size (e.g. inverse of standard error)
• Use statistical test for publication bias → they tend to be a weak statistical test

Question 21

What is a funnel plot?

Answer 21

• A plot of some measure of study size (e.g. standard error of estimate) against measure of effect
• If no publication bias, then the plot will be balanced/symmetrical funnel
• Smaller studies can be expected to vary further from the central effect size
• Publication bias likely to exist if there are few smaller studies with results indicating small or negative measure of effect