Session 6 Reviews of evidence

Question 1

Describe the hierarchy of scientific evidence from strongest to weakest (7)

Answer 1

1. Meta-analyses and systematic reviews
2. RCTs
3. Cohort studies
4. Case-Control Studies
5. Cross Sectional Studies
6. Animal trials and in vitro studies
7. Case reports, opinion papers and letters

Question 2

What should healthcare services and interventions be based on?

Answer 2

best available evidence

Question 3

What should best available medicine be based on?

Answer 3

rigorously conducted research

i.e. primary research studies like RCTs

Question 4

Narrative reviews vs systematic reviews

Assumptions?
Methodology?
Reproducible?
Biased or unbiased?
Subjective or objective?

Answer 4

Narrative:

Assumptions = implicit 
Methodology = opaque 
Reproducible = no
Biased or unbiased = biased
Subjective or objective = subjective

Systematic:

Assumptions = explicit 
Methodology = transparent
Reproducible = yes
Biased or unbiased = unbiased
Subjective or objective = objective

Question 5

Why are systematic reviews extremely credible sources of evidence?

What are the 3 key aspects?

Answer 5

they clearly focus on the question

they include explicit statements about types of:

• study
• participants
• interventions
• outcome measures

they include a systematic literature search and selection of materials

appraisal and synthesis also included

KEY ASPECTS = explicit, transparent and reproducible

Question 6

Describe what a systematic review is vs what a meta-analysis is

Answer 6

systematic review is ‘an overview of primary studies that uses explicit and reproducible methods’

meta-analysis is ‘a quantitative synthesis of the results of two or more primary studies that addressed the same hypothesis in the same way’

Question 7

What are the 4 main purposes of performing a meta-analyses?

Answer 7

1. to facilitate the synthesis of a large number of study results
2. to systematically collate study results
3. to reduce problems of interpretation due to variations in sampling
4. to quantify effect sizes and their uncertainty as a pooled estimate

Question 8

Meta-analysis should have a formal protocol specifying what 4 things?

Answer 8

1. compilation of complete set of studies
2. identification of common variable or category definition
3. standardised data extraction
4. analysis allowing for sources of variation

Question 9

Calculate the odds ratio and 95% CI - what do these indicate?

            survive          die aspirin     [566]             [49] placebo   [557]             [67]

Answer 9

aspirin: odds of surviving = 566:49 = 11.55:1
placebo: odds of surviving = 557:67 = 8.31:1

odds ratio (OR) = 11.55 / 8.31 = 1.39

95% CI = 1.39 divided by or times 1.48 = 0.94 to 2.05

error factor = 1.48

OR indicates greater odds of surviving after MI with aspirin vs placebo

95% CI indicates that null hypothesis (=1.00) is within the range and so the results could be due to chance

Question 10

how would you calculate a pooled estimate odds ratio?

Answer 10

combine 95% CI and OR’s to give a pooled estimate OR using a statistical computer programme

studies are weighted according to their size and uncertainty of their OR’s

narrow CI = greater weight of results

Question 11

How do you interpret a Forest Plot?

squares = 
lines = 
diamond = 
dotted line = 
width of diamond = 
solid line =

Answer 11

squares = individual odds ratios
(size of square is in proportion to the weight given to the study)
lines = 95% CI
diamond = pooled estimate
dotted line = centre of diamond indicates pooled odds ratio
width of diamond = pooled 95% CI
solid line = null hypothesis OR

Question 12

What are the common problems with meta-analyses? 3

Answer 12

Heterogeneity between studies
Variable quality of studies
Publication bias in selection of studies

Question 13

There are two approaches to calculating the pooled estimate odds ratio and its 95% CI. What are they and what do they assume?

(this is to do with heterogeneity between studies and modelling for variation)

Answer 13

Fixed effect model: assumes that the studies are estimating exactly the same true effect size

Random effects model: assumes that the studies are estimating similar, but not the same, true effect size

Question 14

Fixed effect model vs random effects model comparison of:

• Point estimate (e.g. OR) =
• 95% CI =
• Weighting of the studies =

Answer 14

Point estimate (e.g. OR) 
* often similar (but not always!) in both fixed and random

95% CI
* often wider in the random effects than in fixed effects

weighting of the studies

• more equal between the studies in the random effects than in fixed effects
  i. e. greater weighting towards smaller studies

NB: there is a lot of debate over which model is superior!

Question 15

Random effects model can only account for variation but can’t explain it so what could you do to help explain heterogeneity which may prove further insight into the effect of a treatment or exposure? what does this focus on?

Answer 15

sub-group analysis!

looks at study characteristics e.g. year of publication, length of follow up, %female participants

participant profile - where data is analysed by types of participants (e.g. subgroups of males, females, adults, children)

Question 16

Variable quality of studies is an issue in meta-analyses. What can variable quality be due to?
List 4 types of studies that are susceptible to bias and confounding from less to more.

Answer 16

variable quality can be due to:

• poor study design
• poor design protocol
• poor protocol implementation

less susceptible to more susceptible to bias and confounding:
RCT > non-RCT > cohort > case-control

Question 17

What are the two approaches used to try and lessen variable quality of studies?

Answer 17

1. define a basic quality standard and only include studies satisfying this criteria e.g. Cochrane reviews used to include only RCT’s
2. score each study for its quality and then:
   * incorporate the quality score into the weighting allocated to each study during the modelling, so that higher quality studies have a greater influence on the pooled estimate
   * use sub-group analyses to explore differences e.g. high quality studies vs low quality studies

Question 18

When assessing the quality of RCT’s, what are the main components of the scales to assess them?

Answer 18

• allocation methods e.g. randomisation?
• blinding and outcome assessment
• patient attrition e.g. <10% intention to treat
• appropriate statistical analysis

Question 19

What are the reasons and consequences of publication bias in selection of studies?

Answer 19

Studies with statistically significant or ‘favourable’ results are more likely to be published than those studies with non-statistically significant or ‘unfavourable’ results
(this particularly applies to smaller studies)

Any systematic review or meta-analysis can be flawed by such bias
- publication bias leads to a biased selection of studies towards demonstrating an effect

Question 20

What are the methods of identification of publication bias? 3 things!

Answer 20

1. check meta-analysis protocol for method of identification of studies (should include searching and identification of unpublished studies)
2. plot results of identified studies against a measure of their size (e.g. inverse of standard error) i.e. a Funnel Plot
3. Use a statistical test for publication bias (tend to be weak statistical tests)

Question 21

How do you interpret a Funnel Plot

when would publication bias be likely to exist?

Answer 21

= a plot of some measurement of study size (e.g. standard error of estimate) against a measure of effect (e.g. OR)

If no publication bias = plot will be a balanced / symmetrical funnel

Smaller studies can be expected to vary further from the ‘central’ effect size

Publication bias is likely to exist if there are few small studies with results indicating small or negative measure of effect

Question 22

What are the sources of evidence for systematic reviews?

Answer 22

NHS centre for reviews and dissemination

NIHR health technology assessment programme