EBP term 3 module 3 Flashcards
Appraising SRs and meta-analysis
- SRs & meta-analysis are ranked at the top of “evidence
hierarchy” because their conclusions are based on a
synthesis of multiple studies rather than a single trial. - This ranking, assuming the review in question to be of high
quality in its own right, must be determined by the clinician
(You) rather than taken for granted. - Clinicians have to assure that the authors of the review
have done an adequate job of locating, summarizing,
evaluating, and synthesizing the information that guides
clinical decisions. - Potentials for low quality reviews exist due to
- How the review was conducted, and
- How the raw materials (e.g., individual studies) were synthesized.
Evidence on SRs that physical therapists use should FIRST be evaluated with the
following questions that are adapted from “Critical Appraisal Worksheets” developed
by the Oxford Center for Evidence-based Medicine. Diane Jewell (2018) p.342
- did investigators limit the review to high quality studies?
- did implement a comprehensive search and study selection process?
- did the investigators assess the quality of individual studies with standardized processes and or tools?
- did the investigators provide details about the research validity or quality of studies included in the review?
- did the investigators address publication bias?
- if this is a meta analysis, did the investigators use individual subject data in the analysis?
Appraisal of validity of Systematic Reviews (Interventions)
1. Did the investigators limit the review to high-quality studies? (e.g.,
study design being true experimental or quasi-experimental for
intervention studies)
- The above question addresses potential for bias if individual studies
have weak controls in their designs.
Appraisal of validity of Systematic Reviews (Interventions)
2. Did the investigators implement a comprehensive search and study
selection process?
- Addresses potential for selection bias (a non-representative sample)
✓A thorough search requires both electronic and manual efforts to be
successful.
Assessment of Study Credibility – Systematic Reviews
3. Did the investigators assess the quality of individual studies with
standardized processes and/or tools? (e.g., PEDro scale; Cochrane RoB)
- Administered by two independent reviewers; a third reviewer is
needed to resolve disagreement - Addresses potential for various forms of biases in individual studies,
especially related to biases such as
✓“subjects allocation to groups”
✓“outcome assessors blinded to group allocation or not”
✓“loss of subjects leading to imbalance in group characteristics and/or
methods used to deal with the loss”
✓and others to name a few here
Example of Cochrane Risk of Bias
- Random assignment
- Allocation concealment
- Blinding of patients and investigators
- Blinding of assessors
- Incomplete outcome data
- Reporting bias
- Other bias
Assessment of Study Credibility – Systematic Reviews
4. Did the investigators provide details about the research validity
(or quality) of studies included in the review?
- Addresses the need to interpret the review’s results in the
context of quality ratings - Appraising the quality of individual studies is NOT sufficient by
itself; At a minimum, authors should ALSO report level of quality
for each article reviewed. For example: - Cut-off scores for PEDro scale: between 0-3 are considered ‘poor’, 4-5
‘fair’, 6-8 ‘good’, and 9-10 ‘excellent’ study designs. - A SR is likely to be more useful if
✓level of quality of individual studies is reported and
✓results are interpreted in light of these appraisals.
Grading of Recommendations Assessment, Development and Evaluation (GRADE):
rating quality of evidence and strength of recommendations
Grading of Recommendations Assessment, Development and Evaluation
(GRADE)
- Reviews based on RCTs are rated high. This rating can be downgraded
one or more levels, however, by the presence of one or more of the
following factors:
1. High likelihood of bias from study design and implementation
2. Indirectness of population, intervention, control, or outcomes (differences
in PICO across trials)
3. Unexplained heterogeneity or inconsistency of the findings
4. Imprecision or wide confidence intervals
5. Publication bias
Assessment of Study Credibility – Systematic Reviews
5. Did the authors address publication bias?
Publication bias:
i. only studies in English are included in the review
✓Authors are expected to explicitly discuss about it (e.g., no author reads non-
English-language publication)
ii. preference for publication is given to studies showing statistically
significant beneficial treatment effects;
✓the tendency of journal editors to publish studies based on direction and
statistical significance of the outcomes;
✓out of authors’ control;
Raising concerns about an accessibility of ‘unpublished findings’ grey literature:
“composed of nonpublished information, often on websites, in reports,
conference proceedings, theses/dissertations. These sources can be valuable to a
full understanding of the scope of knowledge on the topic”
* Addresses the need to interpret the review’s results in the context of
this potential problem; EX: PRISMA (Eligibility criteria)
✓Specify study characteristics (e.g., PICOS, length of follow-up) and report
characteristics (e.g., years considered, language, publication status) used
Assessment of Study Credibility – Systematic Reviews
6. If this is a meta-analysis, did the investigators use individual patient data in the analysis?
- Meta-analysis may be conducted
- using “summary score” of individual studies or
- using individual patients data (preferred because individual patients data provide more information)
- To obtain them, review authors need to request access from authors of individual studies; Unfortunately, it may not always be accomplished due to logistical challenges.
Study Credibility – Systematic Reviews
Should You Use this Evidence?
7. How confident are you in the research validity of this review?
This checklist refers to adequacy of reporting in each component of the
review, including
* a clear research question,
* criteria for selection of studies,
* a comprehensive search,
* reliability of reviewers, and
* assessing risk of bias.
* For a meta-analysis, use of appropriate
statistical estimates,
* evaluation of heterogeneity and
* publication bias.
* The appraiser is expected to make a
judgment regarding the overall confidence in
the results of the SRs (high, moderate, low,
and critically low) based on the number of
noncritical weaknesses and critical flaws in
the SRs identified by the checklist
Evidence and the Patient/Client’s Perspective
Consider the following questions:
▪ Is the review high quality (e.g., does the design minimize bias)? (Quality)
▪ Are the results important enough to use? (Importance) clinical vs.
statistical
▪ Was your patient/client represented in the review? (Similarity)
▪ Is the element of interest available, practical and safe for application in
the clinical setting? (Practicality)
AND
▪ Patient’s values and preferences
➢ Cost – financial, time, personal
➢ Belief in the value of scientific evidence
➢ Previous experiences
Homogeneity or heterogeneity of findings among
studies leads to
- Either a Quantitative or a Qualitative review
- Meta-analysis (MA)
✓An extension of a systematic review that incorporates statistical
pooling from clinical homogeneous studies
✓The major advantage of MA is increased power obtained by pooling
data from several samples, thereby improving estimates of effect
size and generalization of findings.
Heterogeneity (or homogeneity) can be assessed
using either
✓the ‘eyeball’ test: one looks for overlap of the confidence
intervals of the trials with the summary estimate; or
✓more formally with statistical tests such as I2 statistic
(based on the Cochran Q test). A value of 0% indicates no
heterogeneity, and higher percentages suggest greater
heterogeneity.
Statistics commonly reported for meta-analysis according to
type of research design (or research question)
✓Effect size (ES) or Standardized effect size (intervention studies)
✓Number needed to treat (NNT)
✓Likelihood ratio (LR; more discussion in Diagnostic studies)
✓Odds ratio (OR)
✓Relative risk (RR; aka risk ratio)
Odds ratio and relative risk ratio helps with case control or cohort study!
Meaning of meta-analysis results
Displayed as a forest plot illustrating weighted ES
by individual studies and a cumulative summary
- Result for each study is represented by a tree
plot:
a) Square icon represents effect (e.g., Standardized
Mean Difference – SMD between groups for
intervention study)
b) Whisker (horizontal line) represents confidence
interval (CI) for the SMD
c) Exact numbers for SMD (95%CI) are to the right.
d) Center vertical line is null value (null hypothesis;
no difference between group means or SMD = 0) - Bottom diamond is pooled effect size and the
width is CI for the pooled value. - The study finding is in favor of intervention
Effect size:
the magnitude of the difference
between two mean values; may be standardized
95% Confidence Interval (CI)
a range of scores within which the true score is estimated to lie within a 95% probability 10
effect size notes….
- “a statistical expression of the size of the difference between sample means”
- Calculated as actual differences or standardized differences
Mean Score (Exp) – Mean Score (Control)/Pooled Standard Deviation
Effect size (=a/b)
the magnitude of the difference
between two mean values
➢ Between-group difference (a)
➢ Within-group variability (b)
* strength of Effect size
➢ 0.2 small
➢ 0.5 medium
➢ 0.8 large
Effect Size (standardized) scores
- > 0.8 big treatment effect (no doubt about it!!)
- 0.5 - 0.8 big enough treatment effect that we can
“see it with the naked eye” - 0.2 - 0.49 treatment effect is small enough that we
cannot “see it with the naked eye” - < 0.2 what treatment effect?
Number Needed to Treat (NNT)
✓“the number of subjects treated with an experimental
intervention over the course of a study required to achieve one good outcome or prevent one bad outcome.”
Positive Likelihood Ratio
(LR+)
✓The likelihood that a positive (+) test result
was observed in a person with the disorder v. in a
person without the disorder of interest
- Negative Likelihood Ratio
(LR-)
✓The likelihood that a
negative (-) test result is
observed in a person with
the disorder v. in a person
without the disorder of
interest
Odds Ratio (OR)
a measure of association between exposure and an outcome
✓The “odds” that an individual with the prognostic or
risk factor will develop the outcome of interest
✓Calculated via logistic regression (can also be done via a
2 x 2 table)
Study Results
* Relative Risk or Risk Ratio (RR)
✓Ratio of risk for developing the adverse outcome in
patients with the risk factor versus patients without the risk factor
✓Comparable to odds ratios; the null value is 1 for both OR and RR.
Funnel Plot is for what type of bias
publication bias
Funnel Plot - publication bias
- A funnel plot is used to examine results of MA for the problem of publication bias
- Scatterplot of effect sizes plotted against measure of variability
- Symmetrical distribution indicates both high and low effect sizes with large and small variability
- Asymmetrical distribution implies publication bias (high or low outcomes may be missing)
