Week 11- Evidence Level in Medical Research Flashcards
What is the aim of evidence based medicine/practice?
- Aims to optimize medical practice and decisionmaking by emphasizing the use of evidence from well
designed and conducted research - Medical evidence is classified by its scientific strength,
requiring that only the strongest types of research can
yield strong recommendations
What is the hiearchy of evidence?
Why is evidence-based medicine important?
- High quality research is capable of changing or
improving current clinical practice e.g., encourage
healthcare professionals and systems to use
treatments that have been proven to be both
clinically and cost-effective, while disinvesting from
practice that does not meet these objectives
DATA SYNTHESIS AND EXPERT OPINION DIAGRAM
What is a systematic review?
- A review of a clearly formulated question with
- systematic and explicit methods to identify, select,
and critically appraise relevant research, and - collect and analyse data from the studies that are
included in the review. - Statistical methods (meta-analysis) may or may not
be used to analyse and summarise the results of the
included studies.
What are the steps for undertaking a systematic review?
- Formulate a review question
- Find studies
- Select and Appraise relevant studies
- Summarise and synthesise results
- Interpret and apply results
What should be considered when determining the research question?
- PICO/PECO framework
-> types of Participants
-> types of Intervention (exposure)
-> Comparators (e.g. placebo/other drugs)
-> types of Outcomes - Types of studies (observational/intervention studies)
How does meta-analysis fit into systematic reviews?
It is an optional part of systematic review
What is meta-analysis?
- The statistical component of a systematic review
- It may not always be possible or appropriate to
conduct meta-analysis as part of a systematic
review - Pooling together of results and measure
overall/pooled association/estimates - This will show as if the there is true difference
/association and that the results are not due to
chance
What caused the use of meta-analysis in medical research?
- Formation of Cochrane Collaboration (1993)
spurred growth in meta-analyses in medical
research
Why do we conduct meta-analysis as part of systematic review?
- A single small study may not give very precise
results about the effect of a treatment - Combining results from several studies gives
more precise estimates - May also increase generalizability of results to
different countries/settings
Why is meta-analysis important?
- Researchers put little faith in a single study of an effect, no
matter how good the study and how statistically significant - When many studies were done, someone would write a
narrative (= qualitative) review trying to explain why the
effect was/wasn’t real in the studies - Each study produces a different estimate of the magnitude
- Enlightened researchers now realize that all effects are real
- The aim of research is therefore to get the magnitude of an
effect with adequate precision - Meta-analysis combines the effects from all studies to give
an overall mean effect and other important statistics
When is meta-analysis not performed?
- Do not have data in suitable format
- Studies are poor quality or at risk of bias
- HETEROGENEITY Studies not sufficiently
similar
– differences in: - Populations (eligibility criteria)
- Interventions
- Outcome definitions
- Effect estimates
What is quality assessment (critical appraisal)
- Good quality and poor quality studies
- Many validated quality assessment tools
- Some used more than the others
- Choose a reliable one
- Conclusion we made depends on the quality
of studies
What are examples of quality assessment tools for non-intervention and intervention studies?
- Risk of bias tools for use in systematic reviews of non intervention
studies: - The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised
studies in meta-analyses (cohort and case control studies - NICE Quality appraisal checklists
– Appendix D Methodology checklist: cohort studies
– Appendix E Methodology checklist: case–control studies - Risk of bias tools for use in systematic reviews of intervention
studies: - RoB 2.0 tool (RCTs)
- ROBINS-I tool (Risk Of Bias in Observational Studies focused on Interventions)
What is publication bias?
- A meta-analysis reflects only what’s published.
– But statistically significant effects are more likely
to get published.
– Hence published effects are biased high.
Publication bias = selective publication of studies - investigators fail to report completed studies
- typically those that show ‘no effect’
- suspect when a limited number of small trials all
suggest benefit - big ‘bad pharma’
suppresses evidence
What are the different organisations that provide levels of evidence?
-Oxford Centre for Evidence Based Medicine (2009) (WHAT WE USE)
– National Institute for Health and Care Excellence (NICE) levels of
evidence (2001)
– Australian National Health and Medical Research Council
(NHMRC) levels of evidence (2009)
– United States Preventive Services Task Force (USPSTF) levels of
evidence (1988)
– Grading of Recommendations Assessment, Development and
Evaluation (GRADE) levels of evidence (2004)
What are the levels of evidence (specific)
Explain evidence level 1A
- Types of studies:
– Systematic reviews/meta-analyses of randomized controlled
trials - Strengths:
– Interventional study (exposure assigned not assessed)
– High internal validity (very low random error, bias,
confounding)
– High external validity (assuming representative samples and
low drop out in studies)
– Can prove temporality in associations
– Strongest evidence for inferring causality - Limitations:
– Depend on the quality of included studies; i.e. if original
studies poorly conducted, this cannot be fixed
Explain Evidence Level 1B
- Types of studies:
– High quality individual randomized controlled trials (narrow
confidence intervals, low drop out, random allocation
maintained) - Strengths:
– Interventional study (exposure assigned not assessed)
– High internal validity (low random error, bias, confounding)
– Can prove temporality in associations
– Can infer causality (but with caution - single study) - Limitations:
– Single study, so external validity could be low if the study
sample is too specific (non-representative)
Explain Evidence Level 1C
- Types of studies:
– High quality individual non-randomized/non-controlled trials
(narrow confidence intervals, low drop out) - Strengths:
– Interventional study (exposure assigned not assessed)
– Moderate internal validity (lower bias than in observational
studies)
– Can prove temporality in associations - Limitations:
– Prone to confounding due to lack of randomisation (see
session 11a)
– Single study so external validity could be low if the study
sample is too specific (non-representative)
Explain Evidence Level 2A
- Types of studies:
– Systematic reviews/meta-analyses of cohort studies - Strengths:
– Acceptable internal validity (usually very large
sample, low
random error)
– High external validity (assuming representative samples in
studies)
– Can prove temporality in associations - Limitations:
– Prone to bias and confounding due to methodological
limitations of included studies (observational) (see session
10a)
– Causality cannot be inferred
Explain Evidence Level 2B
- Types of studies:
– High quality individual cohort studies (narrow confidence
intervals, low drop out, accurate assessments, adjustment for
confounding)
– Low quality individual RCTs (wide confidence intervals, high
drop out, random allocation not maintained) - Strengths:
– Acceptable internal validity (large sample, low random error) –
applies only for cohort studies in this case
– Can prove temporality in associations - Limitations:
– Prone to bias and confounding due to methodological
limitations of included studies (see session 10a)
– Causality cannot be inferred
Explain Evidence Level 3A
- Types of studies:
– Systematic reviews/meta-analyses of case-control studies - Strengths:
– Usually very large sample, low random error
– High external validity (assuming representative samples in
studies) - Limitations:
– Low internal validity (bias, confounding) (see session 10a)
– Cannot prove temporality in associations
– Causality cannot be inferred
Explain Evidence Level 3B
- Types of studies:
– High quality individual case-control studies (narrow
confidence intervals, accurate assessments, no serious recall
bias, adjustment for confounding) - Strengths:
– Large sample, low random error - Limitations:
– Low internal validity (bias, confounding) (see session 10a)
– Cannot prove temporality in associations
– Causality cannot be inferred
Explain Evidence Level 4
- Types of studies:
– Case-series (following-up a group of patients who receive a
specific treatment and observe their prognosis)
– Cross-sectional studies (session 10a)
– Low quality cohort and case-control studies (wide confidence
intervals, high bias, affected by confounding) - Strengths:
– Easy to conduct and cheap
– Can generate hypotheses for further higher quality research - Limitations:
– Very low internal validity (random error, bias, confounding)
– Cannot prove temporality in associations (apart from cohort)
– Causality cannot be inferred
– External validity low
Explain Evidence Level 5
- Types of studies:
– Individual expert opinion/report
– Expert committee opinion/report
– Clinical experiences of respected authorities - Strengths:
– Very easy to conduct and usually does not involve a cost
– Could generate hypotheses for further higher quality research - Limitations:
– Do not qualify as scientific research!
– Not based on empirical research findings, just clinical
observations
– Both internal and external validity extremely low
– Definitely cannot infer causality
What are the 4 key factors that affect strength of recommendation?
- Quality of evidence
- Balance desirable/undesirable effects
- Values and preferences
(patients/professionals/paymasters/public) - Cost (utilization of available resources)
Explain the relationship between quality of evidence and strength of recommendation
You can have a weak recommendation, but high quality evidence or strong recommendation, but low quality evidence
What does grade (quality of evidence) include?
- Imprecision 95% CI
- Limitations in study design (bias) quality
- Inconsistency of results heterogeneity
- Publication bias publication bias
- Indirectness of evidence generalisability
What is ‘GRADE’?
- A common, sensible and transparent approach to grading
quality (or certainty) of evidence and strength of
recommendations
Why rate the certainty in the evidence and strength of
recommendations?
– judgments about evidence and recommendations in
healthcare are complex
– clinical actions are likely to differ depending on whether
one concludes that the evidence is low or high
– also need to decide whether any incremental health
benefits are worth the additional costs
– can help to improve communication of practice guidelines
What are the levels of quality acc to GRADE approach?
What factors may decrease quality level of evidence?
What factors may increase quality level of evidence?
