Review Of evidence

Question 1

What is the hierarchy of scientific evidence?

Answer 1

1. Meta-analysis and systematic reviews
2. RCTs
3. Cohort studies
4. Case-control
5. Cross-sectional
6. Animal trials and IV studies
7. Case reports, opinions and letters

Question 2

What does a systematic review consist of?

Answer 2

Clearly focused question

Explicit statement about:

• Types of study
• Types of participants
• Types of interventions
• Types of outcome measures

Systematic literature search

Selection of material

Appraisal

Synthesis

A systematic review is an extremely credible source of evidence because it is EXPLICIT, TRANSPARENT, REPRODUCIBLE

Question 3

What is the difference between systematic review and meta-analysis?

Answer 3

A systematic review is an overview of primary studies that used explicit and reproducible methods

A meta-analysis is a quantitative synthesis of the results of two or more primary studies that addressed the same hypothesis in the same way

Question 4

What is the purpose of a meta-analysis?

Answer 4

To facilitate the synthesis of a large number of study results

To systematically collate study results

To reduce problems of interpretations due to variations in sampling

To quantify effect sizes and their uncertainly as a pooled estimate.

Question 5

What is the criteria for a meta analysis?

Answer 5

Meta-analysis should have a formal protocol which specifies:

Compilation of complete set of studies

Identification of common variable or category definition

Standardised data extraction

Analysis allowing for sources of variation

Question 6

What is a pooled estimate odds ratio?

Answer 6

Odds ratios and their 95% CIs are calculated for all studies in meta-analysis

These are then combined to give a pooled estimate odds ratio using a statistical computer program

Studies are weighted according to their size and the uncertainly of their odds ratio. (Narrower CI = greater weight of results).

Question 7

How do you interpret a forest plot?

Answer 7

Individual odds ration with 95% CI are displaced for each study

Size of square is in proportion to the weight give to the study

The diamond is the pooled estimate with the centre indicating the pooled odds ratio (dotted line) and the width representing the pooled 95% CI.

The sold line is the null hypothesis odds ratio.

Question 8

What are the problems with meta-analysis?

Answer 8

Heterogeneity between studies:

• Modelling for variation (fixed vs random effects model)
• Analysing the variation (sub group analysis)

Variable quality of studies

Publication bias in selection of the studies (only publish those with significant results)/

Question 9

What are the two different approaches to calculating the pooled estimate odds ratio and its 95% CI?

Answer 9

Fixed effect model: assumes that the studies are estimating exactly the same true effect size

Random effects model - assumes that the studies are estimating similar, but not the same true effect size.

Question 10

How does the odds ratio and CI differ between the fixed effect and random effects modelling?

Answer 10

Point estimate (OR) often similar (but not always).

95% CI is often wider for random effects model than in fixed effects model.

The weighting of studies is more equal in the random effects model than in the fixed effects model. This means there is greater weighting towards smaller studies.

Question 11

How do you analyse variation between studies?

Answer 11

Using the random effect or fixed effects model

Random effects modelling can only account for variation but not explain it.

Sub-group analysis can help explain the heterogeneity which may provide further insight into the effect of a treatment or exposure.

• Study characteristics (year of publication, length of follow up, %male / female)
• Participant profile (males / females / adults / children)

Question 12

What could cause variable quality of the studies?

Answer 12

Poor study design
Poor design protocol
Poor protocol implementation

Question 13

What studie are more prone to bias and confounding?

Answer 13

Case control (most prone)
Cohort studies
Non-RCTs
RCTs (least)

Question 14

How do you approach the variable quality of the studies?

Answer 14

Two different approaches:

Define a basic quality standard and only include studies satisfying this criteria e.g. Cochrane reviews used to only include RCTs.

Score each study for its quality and then:

• Incorporate the quality score into the weighting allocated to each study during modelling, so that the higher quality studies have a greater influence on the pooled estimate.
• Use sub-group analysis to explode differences e.g. high quality vs low quality studies.

Question 15

How do you assess the quality of the studies?

Answer 15

RCTs: Many scales are available…

Main components:

• Allocation methods (randomisation?)
• Blinding and outcome assessment
• Patient attrition (<10% and intention to treat)

Who assesses the quality?

• > 1 assessor
• Handling disagreements

Should assessors be blinded to results?
-Sometimes difficult

Question 16

What is publication bias?

Answer 16

This is when there is bias in the studies that are selected to be published.

Studies with statistically significant or favourable results are more likely to be published than those studies with non-statistically significant or unfavourable results. -This particularly applied to smaller studies.

Want systematic review or meta-analysis can be flawed by such bias. -Publication bias leads to a biased selection of studies towards demonstration of effects.

Question 17

How do you identify publication bias?

Answer 17

Check meta-analysis protocol for method of identification of studies -it should include searching for and identifying unpublished studies.

Plot results of identified studies against a standard measure of their size (e.g. inverse of standard error) i.e. a funnel plot.

Use a statistical test for publication bias - they tend to be weak statistical tests.

Question 18

How do you interpret a funnel plot?

Answer 18

A plot of some measure of study size (e.g. standard error of estimate) against measure of effect (odds ratio)

If no publication bias then the plot will be balanced / symmetrical funnel.

Smaller studies can be expected to vary further from the ‘central’ effect size.

Publication bias is likely to exist if there are few small studies with results indicating small or negative measure of effect.