Clinical epidemiology: Systematic reviews of RCTs

Question 1

What are some reasons why there are differences between RCTs?

Answer 1

Differences:

• in people compared
• in care, other than that being investigated
• in the way treatment effects are assessed
• in follow up

Available evidence:

• differences in reporting
• biased selection

Also obsolescence over time, changes in risk profile of patients, missing information
Hopefully not:
- bad science
- fraudulent science

Question 2

What is a systematic review?

Answer 2

Review of all literature on one particular topic using scientific methods:

• clear question being addressed - PICO
• explicit and detailed statement of the methods used
• comprehensive ascertainment of the literature relevant to the review question
• steps to reduce bias in the review process with or without: a statistical technique used to produce a numerical summary of the size of effect with confidence intervals

Question 3

What are the advantages of systematic reviews?

Answer 3

• can be the best source of evidence on the effects and effectiveness of healthcare interventions
• assimilate large amounts of research evidence
• provide reliable unbiased estimates of effect
• increase precision of estimates of effects
• provide information about the generalisability and consistency of effects
• Identify what information is missing
• Useful for making decisions

Question 4

How does the purpose of a narrative review (non-systematic review) differ from a systematic review?

Answer 4

Narrative review:
- background information and current thinking

Systematic review:
- identify, evaluate and summarise current evidence

Question 5

How do the methods of a narrative review (non-systematic review) differ from a systematic review?

Answer 5

Narrative review:
- Not specified

Systematic review:
- specified and reproducible

Question 6

How does the review question of a narrative review (non-systematic review) differ from a systematic review?

Answer 6

Narrative review:
- often general discussion with no stated hypothesis

Systematic review:
- starts with clear question or hypothesis to be tested

Question 7

How does the identification of studies of a narrative review (non-systematic review) differ from a systematic review?

Answer 7

Narrative review:
- well known articles, filling cabinet or selected e.g. medline only

Systematic review:
- planned search strategy for published and unpublished studies

Question 8

How does the study selection for inclusion of a narrative review (non-systematic review) differ from a systematic review?

Answer 8

Narrative review:
- do not usually describe why studies are included/excluded

Systematic review:
- explicit criteria for included and excluded studies

Question 9

How does the quality of assessment and data extraction of a narrative review (non-systematic review) differ from a systematic review?

Answer 9

Narrative review:
- subjective data extraction, do not usually consider study methods and bias

Systematic review:
- objective extraction of data, critical examination of study methods and bias -standard quality assessment method

Question 10

How does the analysis of results of a narrative review (non-systematic review) differ from a systematic review?

Answer 10

Narrative review:
- various; often don’t consider differences between studies

Systematic review:
- Validated methods, assess heterogeneity, meta-analysis

Question 11

What is the first stage of a systematic review?

Answer 11

Define question:
- based on a well defined PICO, is the question answerable?, is there already an up to date well conducted systematic review?

Question 12

What is the second stage of a systematic review?

Answer 12

Write protocol

• why is the review important?
• description of the methods to be used: search strategy, data collection and analysis
• ensures methods and problems are considered beforehand
• Reduces the introduction of bias - prevents cheating as the protocol has to be registered

Question 13

What is the third stage of a systematic review?

Answer 13

Search for evidence/studies
- Comprehensive identification of relevant studies using: bibliographic databases (medline, embase), grey literature, non-english literature, contact key experts/authors

A good review should look at all languages

Question 14

What is the fourth stage of a systematic review?

Answer 14

Select relevant studies from search

• unbiased selection of studies for review
• 2 reviewers independently using pre-defined criteria based on PICO to select relevant studies
• reduce errors and bias - looking at very specific criteria to include/exclude - first you do abstracts then you do full paper

Question 15

What is the fifth stage of a systematic review?

Answer 15

Appraise studies

• unbiased assessment of methodological quality of each study
• two reviews independently using checklist e.g. CASP
• reduce errors and bias

Cochrane database have a manual of how to do a systematic review

Question 16

What is the sixth stage of a systematic review?

Answer 16

Extract, analyse and summarise data

• unbiased extraction of relevant data from studies in review
• 2 reviewers independently using pre-defined approach to identify relevant data in studies
• pre-defined analysis of data
• use of validated appropriate methods of numerical data synthesis (meta-analysis)

Question 17

What is the seventh stage of a systematic review?

Answer 17

Interpret review results

• unbiased interpretation
• considers the results in the context of features and quality of the studies included in the review
• eg. age of trial may mean the comparator out of date

Question 18

What is the final stage of a systematic review?

Answer 18

Discussion and conclusions
- consider implications of review findings in the context of current practice and knowledge and outlines specific future research
Summary of findings

Question 19

In a forest plot, what does the size of the square mean?

Answer 19

Relates to the weight given to the study in the review - very affect by SD = dispersion of results within the study

Question 20

In a forest plot, what does the middle of the square mean?

Answer 20

represents the result e.g. the relative risk for a study

Question 21

In a forest plot, what does the diamond mean?

Answer 21

Summarised result - the width of the diamond represents 95% CI

Question 22

In a forest plot, what does the line in each study mean?

Answer 22

the line represents the 95% CI

Question 23

In a forest plot, what does the mid line mean?

Answer 23

line of no difference between the trial groups

Question 24

What is the I2 used for ?

Answer 24

measuring level of heterogeneity = any variability between studies in a systematic review- could be clinical, methodological, or statistical heterogeneity
<25 = level of difference that doesn’t matter
>75 = worry trials are too different

Question 25

What papers are more likely to be published?

Answer 25

Positive results bias - authors are more likely to submit and editors are more likely to accept positive results than negative or inconclusive results

Question 26

What is outcome reporting bias?

Answer 26

several outcomes in a trial are reported selectively depending on the strength and direction of those results

Question 27

What is language bias?

Answer 27

Positive results and large studies are more likely to be in english
Negative results and smaller studies are less likely to be in english
Sometimes a time lag for non-english studies

Question 28

How can study size influence bias?

Answer 28

Small studies most likely to find small non-significant effects
Small studies that find a large effect size (by chance) are more likely to be published
Small studies with non-significant results are less likely to be published

Question 29

What are funnel plots?

Answer 29

Scatter plot of the treatment effect size against a measure of each study’s size or precision
- effect size = horizontal
- study size = vertical
Subjective assessment of symmetry