Systematic Reviews and Meta-Analysis

Question 1

Describe what is meant by a systematic review

Answer 1

Systematic Review – A review of a clearly formulated question that uses systematic and explicit methods to identify, select, critically appraise relevant research, collect data from the studies, analyse data from included studies.

Question 2

Describe what is meant by a meta-analysis

Answer 2

Meta-analysis - The use of statistical techniques in a systematic review to integrate the results of included studies (matching the eligibility criteria)

Question 3

Can systematic reviews be updated

Answer 3

Yes, depending on how dynamic the field is

Question 4

Why do we conduct systematic reviews

Answer 4

Because of the high volume of data that need to be considered by clinicians and researchers, it has become impossible for the individual to critically evaluate and synthesize the state of current knowledge in many areas. Individual studies usually provide insufficient power to answer a research question.
Furthermore, multiple studies of the same research question often lead to inconsistent or even opposite results. In order to provide more generalizable conclusions, researchers can conduct a systematic review of the primary studies on a particular research question to provide a comprehensive summary of our knowledge at the time of the review.
In other words, a systematic review is ‘a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review.’

Question 5

What are the advantages of a systematic approach

Answer 5

Transparent process because of the explicit methods in identifying and rejecting studies.
A meta-analysis, if appropriate, will increase the power of the study and enhance the precision of estimates of treatment effects, accounting for sample size, and uncertainties.
Systematic reviews may demonstrate the lack of adequate evidence and thus identify areas where further studies are needed.

Question 6

What are some of the limitations of the data qualities of an individual study

Answer 6

often unable to conclusively answer a research question:
often poor study design or small numbers- low power- false negative results
often look only at a subset of the potential study population (the very old, most severely ill), making the results difficult to generalise

Question 7

What are some of the limitations of the data qualities from multiple studies

Answer 7

Which source to trust when results are different or even diverging?
How to rigorously compare several studies using different protocols?

Question 8

Why do we use evidence-based medicine

Answer 8

Decision making process
Based on clinical expertise
personal experience/preferences, concerns, expectations, values.
shared experience through education, senior colleagues and research evidence.
Based on patients’ values and preferences

Question 9

Describe, simply, how we conduct the review

Answer 9

▪ Identification of research ▪ Selection of research ▪ Study quality assessment ▪ Data analysis: tables ▪ Meta-analyses

Question 10

What does assessing the quality and quantity of research require

Answer 10

Efficient searching of data (i.e. literature)

Applying formal rules for critical appraisal of the data sources

Question 11

What is involved in identifying the research

Answer 11

2b: Selection of studies

2c: Quality assessment

Question 12

What is involved in data analysis

Answer 12

3b: Data visualization

3c: Reporting & dissemination

Question 13

Describe how we plan the systematic review

Answer 13

Planning the review
Need to specify the question to be addressed, usually framed around:
The population
The intervention/comparison
The outcomes
The study designs
PICOS

Question 14

What does population involve

Answer 14

Types of study participants

Question 15

Describe what outcomes may involve, in the case of the influenza study

Answer 15

Primary outcomes
Clinical: Symptomatic influenza and influenza-like illness : Numbers of cases, disease complications, working days lost
Harms: Number and seriousness of systemic adverse effects (malaise, nausea, fever, arthralgia, rash, headache) and serious signs (e.g. neurological harms)
Maternal outcomes: abortion -spontaneous, internal, fetal death, stillbirth-, preterm birth (less than 37 weeks), maternal death
Neonatal outcomes: congenital malformations (minor and major), neonatal death

Secondary outcomes
Local adverse effects : induration, soreness and redness at the site of inoculation

Question 16

Describe the study designs in the influenza trial

Answer 16

Randomised controlled trial (RCT) or quasi-RCT
Comparing influenza vaccines in humans with placebo or no intervention
Or comparing types, doses or schedules of influenza vaccine

Comparative non-randomised studies if
reported on serious adverse effects, such as Guillain-Barré syndrome or oculo-respiratory syndromes
or on efficacy of vaccine administration during pregnancy

Question 17

Describe the steps involved in identifying research

Answer 17

Clearly defined search criteria
MeSH (Medical Subject headings) and free text words in combination with Boolean operators

Search the published medical literature
Electronic databases such as Cochrane Central Register of Trials, Medline, EMBASE

Search other sources
Reference lists/citation searches
Conference proceedings/grey literature
Contacting established researchers in the field to identify unpublished studies

Question 18

Describe the methods we can use to identify research

Answer 18

Electronic searches
Cochrane Central Register of Controlled Trials (CENTRAL)
MEDLINE (PubMed)
EMBASE
WHO International Clinical Trial Registry Platform
ClinicalTrials.gov
Other resources
bibliographies of retrieved articles
hand searched journal Vaccine from its first issue to the end of 2009
wrote to manufacturers and first or corresponding trial authors of studies in the review (listed in the review)

Question 19

Describe report selection

Answer 19

Reports selected based on clearly defined inclusion and exclusion criteria.

Question 20

Describe data extraction

Answer 20

Data extracted incl. methodological quality of studies; study design; description of setting; characteristics of participants; description of vaccines (content and antigenic match); description of outcomes; publication status; date of study; location of study.

Question 21

How can we assess study quality and what do we assess

Answer 21

May be assessed according to recognized or user-defined criteria
Quality criteria should assess various biases in study design:
Selection bias
Measurement bias (in exposure and/or outcome assessment)
Attrition bias/loss to follow-up

Question 22

When should we assess study quality

Answer 22

Preferably assessed before study results known, and ideally assessed independently by more than 1 assessor

Question 23

Describe the forest plot

Answer 23

most common way of presenting the results from a meta-analysis.
graphical representation of the results from each study included in a meta-analysis, together with the combined meta-analysis result.

Question 24

How do we interpret forest plots

Answer 24

Left hand side- favours vaccine
Right hand side- risk of vaccine
Middle no effect
All vs placebo
Diamond summarises all the data
Each studies weighed in meta-analysis

Question 25

What is the primary unit of a meta-analysis study

Answer 25

In a meta-analysis the studies themselves are the primary units of analysis as there is usually no access to raw data from each individual study

Question 26

What does the approach of meta-analysis allow us to do

Answer 26

Meta-analyses combine the published estimates of effect from each study to generate a pooled risk estimate. This approach means that:
More subjects can be included than any single constituent study, producing a more reliable and precise estimate of effect;
Differences (heterogeneity) between published studies can be identified and explored.
If the studies are too heterogeneous, it may be inappropriate, even misleading to statistically pool the results from separate studies.

Question 27

What does a meta-analysis involve

Answer 27

Effect estimates are abstracted from the selected studies

To calculate a weighted average of effects across all studies

Question 28

How do we weigh the different studies in a meta-analysis

Answer 28

Most weight to informative studies (often large studies with precise effect estimates)
Least weight to less informative studies (often smaller studies with imprecise effect estimates)

Question 29

Describe the reporting and dissemination of results

Answer 29

Study details tabulated in a meaningful way 
Should include details of: 
the populations 
the interventions/comparison 
the outcomes 
the study design 
Often includes a summary of findings.

Question 30

What guidelines can we use for reporting

Answer 30

STREGA, STROBE, STARD, SQUIRE, MOOSE, PRISMA, GNOSIS, TREND, ORION, COREQ, QUOROM, REMARK… and CONSORT

Question 31

What are some of the limitations of a systematic review/meta-analysis

Answer 31

Publication bias
Inconsistency of results (heterogeneity)
Low study quality

(Incompleteness of the review: not all published data included because of other language, no access to report, errors, etc)
(Low number of studies)
(Lack of generalisability)

Question 32

Describe publication bias

Answer 32

only a subset of the relevant data is available
Null or non significant findings (esp. in small studies) are less likely to be reported/published than statistically significant findings

=> Published studies may not be truly representative of all valid studies undertaken, and this bias may distort meta-analyses and systematic reviews on which evidence-based medicine relies

Question 33

Describe the graphical representation of publication/selection bias

Answer 33

The Funnel plot
Each study is a point; the position is determined by the study result –odds ratio- and the study precision (sample size)
No bias: symmetric about the mean effect and shaped like an upside down funnel
Bias: asymmetric, missing lower right or left hand corner

Question 34

How can studies differ

Answer 34

Populations 
Interventions/exposure 
Outcomes 
Study design 
Clinical differences 
Methodological differences 
Unknown study characteristics

Question 35

What is Tau squared an estimate of

Answer 35

Tau2: estimate of between-study variance based on random-effect model

Question 36

What is chi squared an estimate of

Answer 36

Chi2: test of statistical significance for heterogeneity (low power to detect existing inconsistency )

Question 37

What is I squared a measurement of and what are the cut-offs

Answer 37

measure or index of heterogeneity 
Suggested cut-offs:
I2 = 0%   -> no heterogeneity
I2 = 25% -> low heterogeneity
I2 = 50% -> moderate heterogeneity
I2 = 75% -> high heterogeneity

Arbitrary, except for 0.

I2 can never reach 100% and values above 90% are very
rare.

Question 38

What do interpretations of I squared depend on

Answer 38

Interpretation depends on number of studies, effect size

Question 39

Describe quality of evidence

Answer 39

The real impact of biases could not be determined for about 70% of the included studies (e.g. insufficient reporting details, very different scores among the items evaluated).
17.5% of the included studies (mainly cohorts) had a high risk of bias.
Around 15% of the included studies were well designed and conducted.

Question 40

How can we explore sources of heterogeneity

Answer 40

There are several methods to explore sources of heterogeneity
One method is to analyse different sub-groups and examine whether results differ (e.g. age groups, groups defined by type of vaccine, etc.)

Other methods: meta-regression, sensitivity analysis

Question 41

What are the advantages of systematic reviews and meta-analyses

Answer 41

Generate a pooled overall risk estimate
Produce a more reliable and precise estimate of effect
Explore differences (heterogeneity) between published studies.
Identify whether publication bias is occurring.

Question 42

How do we critically appraise a systematic review

Answer 42

Was a clear, predefined question addressed?
In terms of populations, interventions/exposures, outcomes and study designs?
Was a comprehensive search for relevant literature carried out?
Databases, grey literature; time frame; appropriate inclusion/exclusions; languages; duplicate & independent assessment of literature?
Was methodological quality of each study assessed appropriately?
Quality used as inclusion criteria? Quality measures appropriate? Heterogeneity due to quality?
Was heterogeneity (consistency of results) explored?
Heterogeneity due to populations, interventions/exposures, outcomes and study designs?
How credible is the evidence?
Strengths and weaknesses of evidence? Evidence from high quality studies? Impact on clinical practice? Applicability to other populations (external validity)
Check guidelines for reporting (e.g. PRISMA)

Question 43

How do we critically appraise a meta-analysis

Answer 43

Was heterogeneity explored?
Sub group analyses with respect to sub groups of populations, interventions/exposures, outcomes, study designs, study quality.
Was publication bias an issue?
Evidence for ‘missing’ studies? What impact might this have had on the pooled estimate?
Was it appropriate to pool the studies?
Were studies sufficiently homogeneous for to be pooled?
Was the appropriate model used to pool effect estimates?
Fixed versus random effects model.
Did different sub groups of studies give similar results?
Were results consistent across sub-groups? How generalizable are the findings, are there new hypotheses that should be explored?

Question 44

Describe PRIMSA

Answer 44

Preferred Reporting Items for Systematic Reviews and Meta Analyses is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses adopted by many scientific journals. PRISMA focuses on the reporting of reviews evaluating randomized trials, but can also be used as a basis for reporting systematic reviews of other types of research, particularly evaluations of interventions.

Question 45

Describe stage 1 of a systematic review

Answer 45

Planning the review – The authors need to clearly define the research question to be addressed. This question is usually framed around the definition of study participants, intervention (exposure), outcomes and study designs of interest.

Question 46

Describe stage 2 of a systematic review

Answer 46

Identification of research – This requires clearly defined search criteria and a thorough search of all published literature (including exhaustive searches of reference lists, conference proceedings and contact with researchers in the field).
Selection of studies – Inclusion and exclusion criteria should be defined a priori; these are likely to be based on factors such as study design, year, sample size, completeness of information, study quality etc.
Study quality assessment – Study quality can be assessed against recognized or user-defined criteria, usually to establish whether various biases are likely to exist in the in study (e.g. selection bias, measurement bias, attrition bias/loss to followup).

Question 47

Describe stage 3 of a systematic review

Answer 47

Reporting and dissemination – Study details need to be abstracted from each eligible study along with the effect estimate (or details that allow an effect estimate to be calculated). These details need to be tabulated in a meaningful way, including, where appropriate, details of populations, interventions/exposure, outcomes and study design, and a summary of the findings. The last step consists in estimating an overall effect by combining the data, if a meta-analysis is deemed appropriate.

Question 48

Describe the overall estimate of a forest plot

Answer 48

The overall estimate from the meta-analysis is usually shown as a diamond at the bottom of the plot. The centre of the diamond and dashed line corresponds to the summary effect estimate; the width of the diamond represents the confidence interval around this estimate.

Question 49

How else can heterogeneity be explored

Answer 49

Heterogeneity can be explored using Galbraith (radial) plots.
But remember, if too much heterogeneity exists, it might not be appropriate to pool the studies.

Question 50

Describe Cochrane

Answer 50

Cochrane (http://www.cochrane.org) started as an organization involving a large number of international researchers and clinicians to organize medical research information in a systematic way, in order to facilitate the choices that health professionals, patients, policy makers and others face in health interventions according to the principles of evidence-based medicine. Protocols developed and the Cochrane Database of Systematic Reviews (http://www.cochranelibrary.com/) – a database of systematic reviews and meta-analyses – set many standards in the field.