Systemic Reviews Of RCTs Flashcards
Los
• Revisit framework for using population based research in clinical practice (Assess-Access-Appraise-Act)
• Introduce the need for and importance of systematic reviews in healthcare
• Discover what systematic reviews are
• Learn where to find systematic reviews on the
effectiveness of a treatment (Access)
• Understand key methodological aspects of a systematic review on the effectiveness of a treatment (Appraise – validity)
• Consider numerical results of a systematic review on effectiveness (Appraise – results)
• Understand what to consider with regard to applying findings (Act)
What are the key steps to using population based research to help with clinical decisions
➢ Assess
– Formulate a clear clinical question (PICO)
– Identify the study design that would provide the most
accurate answer
➢ Access
– Search for and retrieve the research
➢ Appraise
– Identify the strengths and weaknesses of the research you find and understand what the results mean
➢ Act
– Apply the evidence, taking into account its limitations, alongside other information eg patient preference
Why should we not use only one trial for research
There are differences between trials
Many trails available
Why may there be differences between RCTs
• chance
• differences in people
• differences in intervention / comparator
• differences in care other than that being investigated
• differences in the way treatment effects are assessed
• differences in follow-up
• reporting of the available evidence
• selection from the available evidence
What is publication bias
• When the publication of research results depends on their nature and direction.
– Positive results bias
- outcome reporting bias
What is positive result bias
– Positive results bias - when authors are more likely to submit, or editors accept, positive than negative or inconclusive results
What is outcome reporting bias
– Outcome reporting bias - when several outcomes within a trial are measured but are reported selectively depending on the strength and direction of those results.
What are the issues with using trails
• Large Volume of Information
• Low-quality information drowns out high-quality information?
• Cannot find/read the right evidence for my/my patient’s/my organisations specific needs
• Must critically evaluate the methodological rigor and statistical analyses of a study,
What is the definition of review and its benefits
Any attempt to summarise information
• Volume of literature condensed
• New information made accessible
• Some reviews are extremely good and have taken considerable time to find all the relevant information and review the topic well
What is a systematic review
the research method generally used to synthesis
multiple pieces of research evidence on the effectiveness of a treatment
(and other types of research question)
• Review of all literature
• On one particular topic
• using scientific methods
Top of the “hierarchy of evidence”
Features of a systematic review
• Clear question being addressed by the review
• An explicit and detailed statement of the methods used
• Comprehensive ascertainment of the literature relevant to the review question
• Steps to reduce bias in the review process /documenting of any bias
Can be with or without:
• An appropriate numerical summary of the size of effect with confidence intervals
What are the stages of a systematic review
Define question (pico) - may need to reformulate
Write a protocol
Search for evidence/ studies
Select Relevant Studies from Search
Appraise Studies
Extract, Analyse & Summarise Data
Interpret Review Results
Discussion & Conclusions
How are components of a question clearly defined
Through pico
P How would I describe the group of patients?
I What main intervention am I considering?
C What is the main alternative
intervention?
O What outcome(s) are important?
- context
Why is a protocol written
To describe methods
Ensure methods and problems are considered
Avoids introducing bias
How to select relevant studies from search
Unbiased selection of studies for
review.
Two reviewers independently using pre-defined criteria (based on question(PICO)) to select relevant studies from the search Reduce errors
How to extract, analyse and summarise data
Two reviewers independently using pre defined approach to identify relevant data in studies
Predefined analysis of data
Use of validated appropriate methods of numerical data synthesis (meta analysis)
Pre defined exploration of possible effect modifiers
What does a systematic review essentially do
Combine data from similar studies to produce a Summary Effect of Intervention (I) vs Control (or Comparator) (C) on each Outcome (1,2,…O) in the specified Population (P)
Can do this for many outcomes
To give stronger and more precise measure of te intervention
What will a meta analysis contain
Will have a list of:
Study
Intervention
Comparator
Forest Plot
Relative risk
What is the central horizontal line in forest plots
If box on the line - Signifies No difference in the groups of the trials for the outcome
Intervention vs comparator
Position of horizontal line depends on metric
Relative risk or odds ratio = 1
Mean difference = 0
What is on the x axis
On each side of the x axis
Should be intervention and comparator
To see which results favour which
What do squares represent on the forest plot
Middle of square represent the results
Each square has horizontal line through it which represents confidence interval
Size of square relates to weight the study is given in the meta analysis e.g. bigger sample size = bigger square
What is the diamond in the forest plot
Central position is the Summarised result from all the data in the studies
Width is confidence interval around summary estimate
What should be addressed in discussions and conclusions
Considers implications of the review findings in the context of current practice and knowledge and outlines specific future research
Summary of findings
What is the difference btw meta analysis and systematic reviews
• Systematic review:
– explicit methods to identify and review all of the literature on a specific topic in order to answer a specific question.
• Meta-analysis:
– statistical technique(s) used to combine the quantitative results from a number of studies
Not all systematic reviews contain a meta analysis - sometimes too much variation btw the studies to combine them together
Advantages of systematic reviews
• Can be the best source of evidence on the effects and effectiveness of healthcare interventions
• Assimilation of large amounts of research evidence
• Provide reliable unbiased estimates of effect
2 independent reviewers
• Increase precision of estimates of effects
Width of diamond small. Narrower confidence intervals around summary estimate. Increases precision.
• Provide information about the generalise ability and consistency of effect
• Identify what information is missing
• Useful for making decisions
What do systematic reviews prevent
Unnecessary repetition. Stops patients going into unnecessary trails.
The more studies added to the review the narrower the confidence intervals get. Increased precision.
But still the same results and conclusion
Therefore need systematic review before any new research is done.
Where to find systematic reviews
Epistemonikos
– Systematic review records from many sources including Pubmed/MEDLINE, Embase,
Cochrane Database of Systematic Reviews
Cochrane library
– Cochrane Database of Systematic Reviews (CDSR) - includes all Cochrane Reviews (and
protocols) prepared by Cochrane Review Groups.
MEDLINE (and/or EMBASE) (via FindIt)
– need to use a “filter” for reviews
NHS evidence
What is critical appraisal
• The systematic identification of strengths and weaknesses of pieces of information &
• The application of that information within the limits identified
Broken down into internal and external validity
What is internal validity
extent to which a research study establishes a trustworthy cause-and-effect relationship.1 This type of validity depends largely on the study’s procedures and how rigorously it is performed.
What is external validity
– The degree to which the reviews conclusions apply to other persons in other places and at other times
Internal validity needs to be strong before considering the strength of external validity
What checklist is used for critical appraisal
CASP
Three part process:
1. Is the review trustworthy / valid (internal validity)? Screening questions
Detailed questions
2. What were the review’s results ?
3. How will the results help (external validity)? Relevance
What are the screening questions for critical appraisal
- Did the review ask a clearly-focused question?
- Did the review include the right type of study?
- Did the reviewers try to identify all relevant studies?
- How did the reviewers decide which studies to include and did they assess the quality (validity) of the included studies?
5.If the results of the studies have been combined together (e.g. meta-analysis) was it reasonable to do so?
6.How are the results presented and what is the main result?
7.How precise are these results?
8.Can the results be applied to the local population? - Were all important outcomes considered
- Should policy or practice change as a result of the evidence
contained in this review?
Did the review ask a clearly-focused question?
Consider if the question is ‘focused’ in terms of:
– the population studied
– the intervention given
– the comparator
– the outcomes considered
Did the review include the right type of study?
Consider if the included studies:
– address the review’s question
– have an appropriate study design
Did the reviewers try to identify all relevant studies?
Consider:
– which bibliographic databases were used
– if there was follow-up from reference lists
– if there was personal contact with experts
– if the reviewers searched for unpublished studies
– if the reviewers searched for non-English-language studies
How did the reviewers decide which studies to include and did they assess the quality (validity) of the included studies?
Consider:
– if a clear, pre-determined strategy was used to determine which studies were included in the review and how their quality was assessed
Look for:
– defined selection criteria (based on PICO)
– defined validity process (e.g using CASP checklist)
– more than one assessor
If the results of the studies have been combined together (e.g. meta-analysis) was it reasonable to do so?
Consider whether:
– the studies have similar characteristics
– the results of each study are clearly displayed – the results were similar from study to study
Heterogeneity (difference)
Methodological – different study designs combined
Clinical – different patient characteristics, settings, interventions
Statistical – differences in reported effect size between the trials – look for statistical tests (Chi2, I2,)
– the reasons for any variations in results are discussed
How are the results presented and what is the main result?
Consider:
– how the results are expressed (e.g. relative risk, mean difference etc.)
– how large this size of result is and how meaningful it is
– how you would sum up the bottom-line result of the review in one sentence
How precise are these results?
Consider:
– if a confidence interval were reported. Would your decision about whether or not to use this intervention be the same at the upper confidence limit as at the lower confidence limit?
– if a p-value is reported where confidence intervals are unavailable
On the forest plot = width of the diamond/ horizontal line through the square
Can the results be applied to the local population?
Consider whether:
– the population sample covered by the review could be different from your population in ways that would produce different results
–your local setting differs much from that of the review – you can provide the same intervention in your setting
Were all important outcomes considered?
Consider outcomes from the point of view of the: – individual
– policy makers and professionals
– family/carers
– wider community
Should policy or practice change as a result of the evidence
contained in this review?
Consider:
– whether any benefit reported outweighs any harm and/or cost
– if this information is not reported can it be filled in from elsewhere?
