Systematic Reviews and Meta-Analysis Flashcards
Describe what is meant by a systematic review
Systematic Review – A review of a clearly formulated question that uses systematic and explicit methods to identify, select, critically appraise relevant research, collect data from the studies, analyse data from included studies.
Describe what is meant by a meta-analysis
Meta-analysis - The use of statistical techniques in a systematic review to integrate the results of included studies (matching the eligibility criteria)
Can systematic reviews be updated
Yes, depending on how dynamic the field is
Why do we conduct systematic reviews
Because of the high volume of data that need to be considered by clinicians and researchers, it has become impossible for the individual to critically evaluate and synthesize the state of current knowledge in many areas. Individual studies usually provide insufficient power to answer a research question.
Furthermore, multiple studies of the same research question often lead to inconsistent or even opposite results. In order to provide more generalizable conclusions, researchers can conduct a systematic review of the primary studies on a particular research question to provide a comprehensive summary of our knowledge at the time of the review.
In other words, a systematic review is ‘a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review.’
What are the advantages of a systematic approach
Transparent process because of the explicit methods in identifying and rejecting studies.
A meta-analysis, if appropriate, will increase the power of the study and enhance the precision of estimates of treatment effects, accounting for sample size, and uncertainties.
Systematic reviews may demonstrate the lack of adequate evidence and thus identify areas where further studies are needed.
What are some of the limitations of the data qualities of an individual study
often unable to conclusively answer a research question:
often poor study design or small numbers- low power- false negative results
often look only at a subset of the potential study population (the very old, most severely ill), making the results difficult to generalise
What are some of the limitations of the data qualities from multiple studies
Which source to trust when results are different or even diverging?
How to rigorously compare several studies using different protocols?
Why do we use evidence-based medicine
Decision making process
Based on clinical expertise
personal experience/preferences, concerns, expectations, values.
shared experience through education, senior colleagues and research evidence.
Based on patients’ values and preferences
Describe, simply, how we conduct the review
▪ Identification of research ▪ Selection of research ▪ Study quality assessment ▪ Data analysis: tables ▪ Meta-analyses
What does assessing the quality and quantity of research require
Efficient searching of data (i.e. literature)
Applying formal rules for critical appraisal of the data sources
What is involved in identifying the research
2b: Selection of studies
2c: Quality assessment
What is involved in data analysis
3b: Data visualization
3c: Reporting & dissemination
Describe how we plan the systematic review
Planning the review Need to specify the question to be addressed, usually framed around: The population The intervention/comparison The outcomes The study designs PICOS
What does population involve
Types of study participants
Describe what outcomes may involve, in the case of the influenza study
Primary outcomes
Clinical: Symptomatic influenza and influenza-like illness : Numbers of cases, disease complications, working days lost
Harms: Number and seriousness of systemic adverse effects (malaise, nausea, fever, arthralgia, rash, headache) and serious signs (e.g. neurological harms)
Maternal outcomes: abortion -spontaneous, internal, fetal death, stillbirth-, preterm birth (less than 37 weeks), maternal death
Neonatal outcomes: congenital malformations (minor and major), neonatal death
Secondary outcomes
Local adverse effects : induration, soreness and redness at the site of inoculation
Describe the study designs in the influenza trial
Randomised controlled trial (RCT) or quasi-RCT
Comparing influenza vaccines in humans with placebo or no intervention
Or comparing types, doses or schedules of influenza vaccine
Comparative non-randomised studies if
reported on serious adverse effects, such as Guillain-Barré syndrome or oculo-respiratory syndromes
or on efficacy of vaccine administration during pregnancy
Describe the steps involved in identifying research
Clearly defined search criteria
MeSH (Medical Subject headings) and free text words in combination with Boolean operators
Search the published medical literature
Electronic databases such as Cochrane Central Register of Trials, Medline, EMBASE
Search other sources
Reference lists/citation searches
Conference proceedings/grey literature
Contacting established researchers in the field to identify unpublished studies
Describe the methods we can use to identify research
Electronic searches
Cochrane Central Register of Controlled Trials (CENTRAL)
MEDLINE (PubMed)
EMBASE
WHO International Clinical Trial Registry Platform
ClinicalTrials.gov
Other resources
bibliographies of retrieved articles
hand searched journal Vaccine from its first issue to the end of 2009
wrote to manufacturers and first or corresponding trial authors of studies in the review (listed in the review)
Describe report selection
Reports selected based on clearly defined inclusion and exclusion criteria.
Describe data extraction
Data extracted incl. methodological quality of studies; study design; description of setting; characteristics of participants; description of vaccines (content and antigenic match); description of outcomes; publication status; date of study; location of study.
How can we assess study quality and what do we assess
May be assessed according to recognized or user-defined criteria
Quality criteria should assess various biases in study design:
Selection bias
Measurement bias (in exposure and/or outcome assessment)
Attrition bias/loss to follow-up
When should we assess study quality
Preferably assessed before study results known, and ideally assessed independently by more than 1 assessor
Describe the forest plot
most common way of presenting the results from a meta-analysis.
graphical representation of the results from each study included in a meta-analysis, together with the combined meta-analysis result.
How do we interpret forest plots
Left hand side- favours vaccine Right hand side- risk of vaccine Middle no effect All vs placebo Diamond summarises all the data Each studies weighed in meta-analysis
What is the primary unit of a meta-analysis study
In a meta-analysis the studies themselves are the primary units of analysis as there is usually no access to raw data from each individual study
What does the approach of meta-analysis allow us to do
Meta-analyses combine the published estimates of effect from each study to generate a pooled risk estimate. This approach means that:
More subjects can be included than any single constituent study, producing a more reliable and precise estimate of effect;
Differences (heterogeneity) between published studies can be identified and explored.
If the studies are too heterogeneous, it may be inappropriate, even misleading to statistically pool the results from separate studies.
What does a meta-analysis involve
Effect estimates are abstracted from the selected studies
To calculate a weighted average of effects across all studies
How do we weigh the different studies in a meta-analysis
Most weight to informative studies (often large studies with precise effect estimates)
Least weight to less informative studies (often smaller studies with imprecise effect estimates)
Describe the reporting and dissemination of results
Study details tabulated in a meaningful way Should include details of: the populations the interventions/comparison the outcomes the study design Often includes a summary of findings.
What guidelines can we use for reporting
STREGA, STROBE, STARD, SQUIRE, MOOSE, PRISMA, GNOSIS, TREND, ORION, COREQ, QUOROM, REMARK… and CONSORT
What are some of the limitations of a systematic review/meta-analysis
Publication bias
Inconsistency of results (heterogeneity)
Low study quality
(Incompleteness of the review: not all published data included because of other language, no access to report, errors, etc)
(Low number of studies)
(Lack of generalisability)
Describe publication bias
only a subset of the relevant data is available
Null or non significant findings (esp. in small studies) are less likely to be reported/published than statistically significant findings
=> Published studies may not be truly representative of all valid studies undertaken, and this bias may distort meta-analyses and systematic reviews on which evidence-based medicine relies
Describe the graphical representation of publication/selection bias
The Funnel plot
Each study is a point; the position is determined by the study result –odds ratio- and the study precision (sample size)
No bias: symmetric about the mean effect and shaped like an upside down funnel
Bias: asymmetric, missing lower right or left hand corner
How can studies differ
Populations Interventions/exposure Outcomes Study design Clinical differences Methodological differences Unknown study characteristics
What is Tau squared an estimate of
Tau2: estimate of between-study variance based on random-effect model
What is chi squared an estimate of
Chi2: test of statistical significance for heterogeneity (low power to detect existing inconsistency )
What is I squared a measurement of and what are the cut-offs
measure or index of heterogeneity Suggested cut-offs: I2 = 0% -> no heterogeneity I2 = 25% -> low heterogeneity I2 = 50% -> moderate heterogeneity I2 = 75% -> high heterogeneity
Arbitrary, except for 0.
I2 can never reach 100% and values above 90% are very
rare.
What do interpretations of I squared depend on
Interpretation depends on number of studies, effect size
Describe quality of evidence
The real impact of biases could not be determined for about 70% of the included studies (e.g. insufficient reporting details, very different scores among the items evaluated).
17.5% of the included studies (mainly cohorts) had a high risk of bias.
Around 15% of the included studies were well designed and conducted.
How can we explore sources of heterogeneity
There are several methods to explore sources of heterogeneity
One method is to analyse different sub-groups and examine whether results differ (e.g. age groups, groups defined by type of vaccine, etc.)
Other methods: meta-regression, sensitivity analysis
What are the advantages of systematic reviews and meta-analyses
Generate a pooled overall risk estimate
Produce a more reliable and precise estimate of effect
Explore differences (heterogeneity) between published studies.
Identify whether publication bias is occurring.
How do we critically appraise a systematic review
Was a clear, predefined question addressed?
In terms of populations, interventions/exposures, outcomes and study designs?
Was a comprehensive search for relevant literature carried out?
Databases, grey literature; time frame; appropriate inclusion/exclusions; languages; duplicate & independent assessment of literature?
Was methodological quality of each study assessed appropriately?
Quality used as inclusion criteria? Quality measures appropriate? Heterogeneity due to quality?
Was heterogeneity (consistency of results) explored?
Heterogeneity due to populations, interventions/exposures, outcomes and study designs?
How credible is the evidence?
Strengths and weaknesses of evidence? Evidence from high quality studies? Impact on clinical practice? Applicability to other populations (external validity)
Check guidelines for reporting (e.g. PRISMA)
How do we critically appraise a meta-analysis
Was heterogeneity explored?
Sub group analyses with respect to sub groups of populations, interventions/exposures, outcomes, study designs, study quality.
Was publication bias an issue?
Evidence for ‘missing’ studies? What impact might this have had on the pooled estimate?
Was it appropriate to pool the studies?
Were studies sufficiently homogeneous for to be pooled?
Was the appropriate model used to pool effect estimates?
Fixed versus random effects model.
Did different sub groups of studies give similar results?
Were results consistent across sub-groups? How generalizable are the findings, are there new hypotheses that should be explored?
Describe PRIMSA
Preferred Reporting Items for Systematic Reviews and Meta Analyses is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses adopted by many scientific journals. PRISMA focuses on the reporting of reviews evaluating randomized trials, but can also be used as a basis for reporting systematic reviews of other types of research, particularly evaluations of interventions.
Describe stage 1 of a systematic review
Planning the review – The authors need to clearly define the research question to be addressed. This question is usually framed around the definition of study participants, intervention (exposure), outcomes and study designs of interest.
Describe stage 2 of a systematic review
Identification of research – This requires clearly defined search criteria and a thorough search of all published literature (including exhaustive searches of reference lists, conference proceedings and contact with researchers in the field).
Selection of studies – Inclusion and exclusion criteria should be defined a priori; these are likely to be based on factors such as study design, year, sample size, completeness of information, study quality etc.
Study quality assessment – Study quality can be assessed against recognized or user-defined criteria, usually to establish whether various biases are likely to exist in the in study (e.g. selection bias, measurement bias, attrition bias/loss to followup).
Describe stage 3 of a systematic review
Reporting and dissemination – Study details need to be abstracted from each eligible study along with the effect estimate (or details that allow an effect estimate to be calculated). These details need to be tabulated in a meaningful way, including, where appropriate, details of populations, interventions/exposure, outcomes and study design, and a summary of the findings. The last step consists in estimating an overall effect by combining the data, if a meta-analysis is deemed appropriate.
Describe the overall estimate of a forest plot
The overall estimate from the meta-analysis is usually shown as a diamond at the bottom of the plot. The centre of the diamond and dashed line corresponds to the summary effect estimate; the width of the diamond represents the confidence interval around this estimate.
How else can heterogeneity be explored
Heterogeneity can be explored using Galbraith (radial) plots.
But remember, if too much heterogeneity exists, it might not be appropriate to pool the studies.
Describe Cochrane
Cochrane (http://www.cochrane.org) started as an organization involving a large number of international researchers and clinicians to organize medical research information in a systematic way, in order to facilitate the choices that health professionals, patients, policy makers and others face in health interventions according to the principles of evidence-based medicine. Protocols developed and the Cochrane Database of Systematic Reviews (http://www.cochranelibrary.com/) – a database of systematic reviews and meta-analyses – set many standards in the field.
