Critical appraisal of Randomised Controlled Trials (RCTs) Flashcards
What is meant by critical appraisal?
Carefully and systematically examining research evidence to judge its trustworthiness, validity and relevance in a particular context
Why is critical appraisal necessary for clinicians and policy makers that practise evidence-based medicine?
Critical appraisal determines whether clinical practice/policy should change as result of research evidence
How does a clinician/policy maker come to an overall opinion on the quality and relevance of research evidence?
Answer series of questions
What are the 2 overall steps of critical appraisal of RCT?
Internal validity
External validity
What is meant by critical appraisal of internal validity of RCT?
Extent to which causal conclusion can be reached, and if research evidence can otherwise be explained by chance, bias, confounding
What 3 factors can affect the internal validity of a RCT, so that a causal conclusion is less strong?
Confounding
Bias
Chance
In the intervention and control groups of a RCT, are the groups at the same baseline level?
Intervention and control groups are at same baseline level/equivalent in every aspect except the treatment/intervention
How are equivalent intervention and control groups produced at the start of a RCT, and how is their comparability maintained?
Comparability of intervention and control groups produced at start by randomisation of participants
Throughout RCT, comparability is maintained by good trial design
What method should be used to present a research question, and explain each step of the method?
PICOT method:
P: Participants
I: Intervention
C: Comparator
O: Outcome
T: Timeframe
Give 2 reasons as to why RCT is the best methodology to use when testing an intervention, when examining internal validity?
Testing intervention requires a intervention/active and control group so that effect of intervention can be compared
In turn this makes it easier to determine causality between the outcome and the intervention
What is meant by random allocation?
Participant has known probability of receiving each intervention before being assigned, but assignment to intervention is determined by unpredictable chance process
Give 2 examples of suitable randomised assignment methods in a RCT, when evaluating internal validity?
Random number table
Computerised random number generator
Give 2 examples of non-suitable randomised assignment methods in a RCT, when evaluating internal validity?
Coin toss
Non-random methods eg. D.O.B, hospital number, choosing every alternate participant
What is allocation concealment, and how does it allow internal validity of a RCT?
Allocation concealment is mechanism that prevents researchers from gaining foreknowledge of treatment assignment to participants
Allows internal validity by overall preventing selection bias from occurring during process of randomisation
Give 2 methods of allocation concealment?
Can use third-party randomisation eg. pharmacy assigns participants
Seal drugs in sequentially numbered identical containers according to allocation sequence
What is blinding in a RCT?
Blinding occurs when info is withheld from people involved in the trial who could be influenced by this info
Why is it important to know who is blinded in a RCT, when ensuring internal validity?
Blinding can be preserved throughout RCT, reducing introduction of bias
What are 4 effects of unblinded participants in RCT?
Influences response to intervention
participant uses co-interventions
influence participant compliance
risk of dropping out of trial
What is a co-intervention in a RCT?
Additional treatments received by participant that is not part of RCT intervention, so can cause co-intervention bias
What is the effect of unblinded healthcare workers in a RCT?
Influences provided care to participants
What are 2 effects of unblinded data analysers in a RCT?
Influences choice of analytic strategies
Can select favourable outcomes or time points, which reflect their personal opinion on the intervention
What is the effect of unblinded outcome assessors in a RCT?
Influences recordings of outcomes
What are the 3 types of bias reduced due to blinding in a RCT?
Performance bias: Systematic differences between groups that occur in the study due to knowledge of intervention allocation from researchers, healthcare workers or participants themselves
Attrition bias: type of selection bias that occurs when participants are lost during follow up due to participants dropping out of trial, results in missing data
Detection bias: systematic differences between groups in how outcomes to exposure are assessed eg. how outcome from each group is collected/verified
What is the main difference between allocation concealment and blinding in a RCT?
Allocation concealment eliminates bias during recruitment and assignment processes of randomisation, but blinding prevents bias after randomisation
Give 2 reasons why it is important to ensure that the intervention and control groups of a RCT are similar at the baseline level?
Provides additional evidence that all differences are only due to treatment allocation
Provides additional evidence that findings are causal
Give 3 examples of reported characteristics that should be similar in intervention and control groups in RCT, to establish the baseline level?
demographics, lifestyle, clinical features
If a baseline level is established in a RCT, what is the most likely reason for any differences that still occur between the intervention and control groups apart from treatment allocation?
Chance/random variation
Why is it important that participants are treated equally during follow-up of the RCT, and give 2 examples of when this doesn’t happen?
Differences in follow-up could distort results
Such as performance bias caused when healthcare workers provide better care for one group than the other
Such as detection bias caused when outcome is measured
How can you check that all participants were accounted for at the end of the RCT, to ensure internal validity?
Numbers of analysed participants add up to number of randomised participants
Explain the method that is used to make sure that all participants are analysed according to their intially allocated groups?
Intent-to-treat analysis method is used, where all randomised participants are included in statistical analysis and analysed in group to which they were initially allocated
What is the primary outcome of a RCT and is it relevant to the research question?
Primary outcome is the most relevant variable/main outcome to answer the research question
Why is it important that the primary outcome of a study is pre-specified with a clear definition?
Primary outcome should be prespecified before RCT begins to prevent outcome switching during trial, clear definition is important so that different definitions aren’t analysed with most favourable outcomes reported only (selective reporting)
How can selective reporting of outcomes be prevented in a study?
Primary outcome is clearly defined so that different definitions aren’t analysed with most favourable outcomes reported only (selective reporting)
What is the function of sample size calculation of a study, and how is it calculated?
Sample size calculation determines how many participants are needed for the planned analysis of primary outcome to be informative
Calculated using the expected difference/effect size of primary outcome
What is the relationship between odds ratio and effect size of primary outcome?
Closer the odds ratio is to 1, smaller the effect size of primary outcome
What is the relationship between p values and alpha significance level in determining whether analysis of primary outcome is statistically significant?
If p < alpha (usually 0.05), analysis of primary outcome is statistically significant
What is the relationship between confidence intervals and the line of difference in determining whether analysis of primary outcome is statistically significant?
Line of difference is usually 1
If CI crosses line of difference/1, than there is insufficient evidence to suggest that there is statistical significance
If CI doesn’t cross line of difference/1, than there is sufficient evidence to suggest that there is statistical significance
What is the relationship between the odds ratio and estimate precision of whether the analysis of primary outcome is statistically significant?
The closer the odds ratio is to 1, the more precise the estimate
Eg. Imprecise estimate: with 95% certainty the true population odds ratio is between 0.69 – 1.36 (between 31% reduction and 36% increase in odds)
What 5 factors are needed for sample size calculation?
expected difference for primary outcome
expected baseline events, (usually proportion/rate)
alpha significance level
power (1 - beta) that is usually set at 80-90%
dropout rate
What does external validity determine?
Extent to which research evidence can be generalised to other settings
Eg. Are RCT results likely to be applicable to your patient group
Give 3 examples of characteristics to compare between local patient population and trial population to determine if there is external validity?
Demographics, clinical features and outcome baseline risk
What is the relationship between the eligibility criteria of a RCT and generalisability of the study results, and where in the paper can the eligibility criteria be found?
Inclusion/exclusion criteria (usually found in introduction, method): larger the eligibility criteria for enrolling in the RCT, the better the generalisability of results
What part of the paper shows the outcome baseline risk?
Event rate of control group
What 5 factors should be assessed when deciding if estimated effect of primary outcome is clinically relevant/significant as well as statistically significant?
minimal difference which is of clinical importance
risks
side effects
administration
costs
Give 4 examples of critical appraisal tools?
Cochrane risk of bias tool
CASP checklist
SIGN
NICE
