L22 Systematic Review & Meta-Analysis Flashcards
Define ‘systematic review’.
A review of a clearly formulated question that uses systematic & explicit methods to identify, select, & critically appraise relevant research, & to collect & analyze data from the studies that are included in the review.
- Statistical methods (meta-analysis) may or may not be used to analyze & summarize the results of the included studies
Define ‘meta-analysis’.
The use of statistical techniques to integrate the results of several independent studies included in a systematic review into a single quantitative estimate or summary effect size.
Differentiate between narrative reviews & systematic reviews.
Narrative review | Systematic review
1) Qn: Often broad in scope | Often a focused clinical question
2) Source & search: Not usually specified, potentially biased | Comprehensive sources & explicit search strategy
3) Selection: Not usually specified, potentially biased | Criterion-based selection (i.e. inclusion/exclusion criteria), uniformly applied
4) Appraisal: Variable | Rigorous critical appraisal
5) Synthesis: Often a qualitative review | May or may not include meta-analysis (i.e. quantitative summary provided)
Explain the purpose of meta-analysis in systematic reviews.
1) Increase statistical power via increasing total sample size
2) Improve precision of CI as e.g.
3) Settle controversies arising from apparently conflicting studies or to generate a new hypothesis
4) Answer questions not posed by individual studies (e.g. different ethnic groups affecting association being challenged)
Outline the steps involved in conducting a systematic review w/ or w/o meta-analysis.
1) Formulate the review question & develop the systematic review protocol.
- Use PICO approach to formulate review qn
- Define study selection criteria i.e. inclusion/exclusion criteria to be applied uniformly across databases, based on study design and PICO (e.g. PRISMA-P 2015)
2) Search the literature
- Conduct comprehensive literature search across multiple electronic databases (to be specified in inlcusion/exclusion criteria) and/or hand-searching reference lists of publications
- Formulate search terms (e.g. MESH terms) for electronic search & document search strategy & search period (e.g. from inception)
- Document reasonable limitations placed on search (e.g. language restrictions)
- Study selection based on pre-defined criteria
- Record no. of studies included & excluded at each step of selection process & reasons for exclusion (i.e. PRISMA 2020 flow diagram)
- At least two independent reviewers SHOULD be involved to minimise selection bias!
3) Assess study quality
- At least two independent reviewers SHOULD be involved to minimise selection bias!
- Use of either Jahad scale (RCTs), Cochrane risk-of-bias tool version 2 (RCTs) or Newcastle-Ottawa Scale (NOS) (observational studies) to assess study quality.
4) Abstract data
- At least two independent reviewers SHOULD be involved to minimise selection bias!
- Develop a standardised data abstraction form to assess relevant study characteristics (pre-determined), including quality aspects & all relevant results
5) Analyse data (may include meta-analysis) & interpret results
- Qualitative data synthesis: Tabulation / graphical display of characteristics & results of individual studies
- Quantitative data synthesis (meta-analysis):
(a) Statistical software e.g. Stata, RevMan used
(b) Present data as a Forest plot
(c) Examine heterogeneity among studies
(d) Assess for publication bias
6) Report findings
- Summarise key findings
- Judge validity of findings
- Judge generalizability of findings
- Implications for further research?
- Implications for clinical practice?
What approach is used to frame the review question?
PICO approach: Population Intervention Comparison Outcome
What is the minimum number of independent reviewers required to conduct a systematic review?
Two independent reviewers, to minimise selection bias
Any discrepancies between reviewers to be resolved through discussion until consensus is reached, or unresolved disagreements to be referred to a third investigator for review & resolution (e.g. principal investigator).
Which scales are used to assess the study quality of RCTs in systematic reviews?
Jahad scale & Cochrane risk-of-bias tool version 2 (ROB2)
Why is Cochrane ROB2 better at identifying the study quality in systematic reviews of RCTs, compared to Jahad scale?
Jahad scale is a composite scale ranging from 0 to 5, thus NOT comprehensive enough to identify where the bias lies between RCTs.
- Low quality: 0-2 | High quality: 3-5
In contrast, Cochrane ROB2 is a domain-based scale that allow risk-of-bias judgement for each domain, and thus more comprehensive in identifying gaps in study quality between RCTs.
When is the NOS scale used in systematic reviews?
How is it used?
To assess the quality of observation studies, specifically case-control & cohort studies.
Maximum of 9 stars given based on three domains:
- Selection of cases vs controls / exposed vs unexposed groups (i.e. look for confounding)
- Comparability between cases vs controls / exposed vs unexposed groups (max. 2 stars)
- Asessement of exposure / outcomes respectively (i.e evaluate for information bias)
How do you interpret a Forest plot?
1) Each tree represents a study:
- Square = point estimate of treatment effect
- Horizontal line = 95% CI of point estimate
2) Size of square = weight of each study in meta-analysis = 1 / variance
- Larger square represents study with greatest weight, due to larger sample size and smaller SD
3) Mid-point of diamond = summary effect measure (i.e. pooled result of all studies)
4) Width of diamond = 95% CI of summary effect measure
How do you examine heterogeneity among studies in a systematic review?
Based on three types of heterogeneity:
1) Clinical heterogeneity:
- e.g. due to differences in Tx doses, durations & regimens, in sample populations. in timing & method of measuring outcomes
2) Methodological heterogeneity:
- e.g. due to differences in study designs (e.g. RCT vs observational), study quality (e.g. Jahad, Cochrane ROB2 & ROS)
3) Statistical heterogeneity:
- i.e how well do CIs of studies overlap with each other & the summary effect measure? (e.g. via visual inspection of Forest plot)
- Statistical tests: Cochrane’s Q test & I^2 statistic
(a) Cochrane chi-square test of heterogeneity: P < 0.1 means significant statistical heterogeneity
(b) I^2 statistic represent proportion of total variance due to between-study variability (i.e. = between-study variability / total variance x 100%): 0% indicates no observed heterogeneity & larger values shows increasing heterogeneity
Discuss the advantages and disadvantages of using random-effects model and fixed-effects model of meta-analysis in a systematic review.
Random-effects model:
(+) Accounts for between-study variability & within-study variability
(-) Gives greater weightage to smaller studies, compared to fixed-effects model, which may result in less precise summary effect measure.
Fixed-effects model:
(+) Gives less weightage to smaller studies, compared to random-effects model, which may result in more precise summary effect measure.
(-) Does NOT accounts for between-study variability
How do you examine for publication bias among studies in the meta-analysis of a systematic review?
Visual inspection of a funnel plot OR Egger’s test for funnel plot asymmetry (p < 0.05 means asymmetric i.e publication bias present)
Why do you examine for publication bias among studies in the meta-analysis of a systematic review?
Meta-analyses are subjected to publication bias (sub-category of selection bias):
- Studies with negative results are less likely to be published.
- Published studies tend to give positive results.
- Results from meta-analyses thus may overstate a treatment effect.
