Critical appraisal of Randomised Controlled Trials (RCTs)

Question 1

What is meant by critical appraisal?

Answer 1

Carefully and systematically examining research evidence to judge its trustworthiness, validity and relevance in a particular context

Question 2

Why is critical appraisal necessary for clinicians and policy makers that practise evidence-based medicine?

Answer 2

Critical appraisal determines whether clinical practice/policy should change as result of research evidence

Question 3

How does a clinician/policy maker come to an overall opinion on the quality and relevance of research evidence?

Answer 3

Answer series of questions

Question 4

What are the 2 overall steps of critical appraisal of RCT?

Answer 4

Internal validity

External validity

Question 5

What is meant by critical appraisal of internal validity of RCT?

Answer 5

Extent to which causal conclusion can be reached, and if research evidence can otherwise be explained by chance, bias, confounding

Question 6

What 3 factors can affect the internal validity of a RCT, so that a causal conclusion is less strong?

Answer 6

Confounding
Bias
Chance

Question 7

In the intervention and control groups of a RCT, are the groups at the same baseline level?

Answer 7

Intervention and control groups are at same baseline level/equivalent in every aspect except the treatment/intervention

Question 8

How are equivalent intervention and control groups produced at the start of a RCT, and how is their comparability maintained?

Answer 8

Comparability of intervention and control groups produced at start by randomisation of participants

Throughout RCT, comparability is maintained by good trial design

Question 9

What method should be used to present a research question, and explain each step of the method?

Answer 9

PICOT method:
P: Participants
I: Intervention
C: Comparator
O: Outcome
T: Timeframe

Question 10

Give 2 reasons as to why RCT is the best methodology to use when testing an intervention, when examining internal validity?

Answer 10

Testing intervention requires a intervention/active and control group so that effect of intervention can be compared

In turn this makes it easier to determine causality between the outcome and the intervention

Question 11

What is meant by random allocation?

Answer 11

Participant has known probability of receiving each intervention before being assigned, but assignment to intervention is determined by unpredictable chance process

Question 12

Give 2 examples of suitable randomised assignment methods in a RCT, when evaluating internal validity?

Answer 12

Random number table

Computerised random number generator

Question 13

Give 2 examples of non-suitable randomised assignment methods in a RCT, when evaluating internal validity?

Answer 13

Coin toss

Non-random methods eg. D.O.B, hospital number, choosing every alternate participant

Question 14

What is allocation concealment, and how does it allow internal validity of a RCT?

Answer 14

Allocation concealment is mechanism that prevents researchers from gaining foreknowledge of treatment assignment to participants

Allows internal validity by overall preventing selection bias from occurring during process of randomisation

Question 15

Give 2 methods of allocation concealment?

Answer 15

Can use third-party randomisation eg. pharmacy assigns participants

Seal drugs in sequentially numbered identical containers according to allocation sequence

Question 16

What is blinding in a RCT?

Answer 16

Blinding occurs when info is withheld from people involved in the trial who could be influenced by this info

Question 17

Why is it important to know who is blinded in a RCT, when ensuring internal validity?

Answer 17

Blinding can be preserved throughout RCT, reducing introduction of bias

Question 18

What are 4 effects of unblinded participants in RCT?

Answer 18

Influences response to intervention

participant uses co-interventions

influence participant compliance

risk of dropping out of trial

Question 19

What is a co-intervention in a RCT?

Answer 19

Additional treatments received by participant that is not part of RCT intervention, so can cause co-intervention bias

Question 20

What is the effect of unblinded healthcare workers in a RCT?

Answer 20

Influences provided care to participants

Question 21

What are 2 effects of unblinded data analysers in a RCT?

Answer 21

Influences choice of analytic strategies

Can select favourable outcomes or time points, which reflect their personal opinion on the intervention

Question 22

What is the effect of unblinded outcome assessors in a RCT?

Answer 22

Influences recordings of outcomes

Question 23

What are the 3 types of bias reduced due to blinding in a RCT?

Answer 23

Performance bias: Systematic differences between groups that occur in the study due to knowledge of intervention allocation from researchers, healthcare workers or participants themselves

Attrition bias: type of selection bias that occurs when participants are lost during follow up due to participants dropping out of trial, results in missing data

Detection bias: systematic differences between groups in how outcomes to exposure are assessed eg. how outcome from each group is collected/verified

Question 24

What is the main difference between allocation concealment and blinding in a RCT?

Answer 24

Allocation concealment eliminates bias during recruitment and assignment processes of randomisation, but blinding prevents bias after randomisation

Question 25

Give 2 reasons why it is important to ensure that the intervention and control groups of a RCT are similar at the baseline level?

Answer 25

Provides additional evidence that all differences are only due to treatment allocation

Provides additional evidence that findings are causal

Question 26

Give 3 examples of reported characteristics that should be similar in intervention and control groups in RCT, to establish the baseline level?

Answer 26

demographics, lifestyle, clinical features

Question 27

If a baseline level is established in a RCT, what is the most likely reason for any differences that still occur between the intervention and control groups apart from treatment allocation?

Answer 27

Chance/random variation

Question 28

Why is it important that participants are treated equally during follow-up of the RCT, and give 2 examples of when this doesn’t happen?

Answer 28

Differences in follow-up could distort results

Such as performance bias caused when healthcare workers provide better care for one group than the other

Such as detection bias caused when outcome is measured

Question 29

How can you check that all participants were accounted for at the end of the RCT, to ensure internal validity?

Answer 29

Numbers of analysed participants add up to number of randomised participants

Question 30

Explain the method that is used to make sure that all participants are analysed according to their intially allocated groups?

Answer 30

Intent-to-treat analysis method is used, where all randomised participants are included in statistical analysis and analysed in group to which they were initially allocated

Question 31

What is the primary outcome of a RCT and is it relevant to the research question?

Answer 31

Primary outcome is the most relevant variable/main outcome to answer the research question

Question 32

Why is it important that the primary outcome of a study is pre-specified with a clear definition?

Answer 32

Primary outcome should be prespecified before RCT begins to prevent outcome switching during trial, clear definition is important so that different definitions aren’t analysed with most favourable outcomes reported only (selective reporting)

Question 33

How can selective reporting of outcomes be prevented in a study?

Answer 33

Primary outcome is clearly defined so that different definitions aren’t analysed with most favourable outcomes reported only (selective reporting)

Question 34

What is the function of sample size calculation of a study, and how is it calculated?

Answer 34

Sample size calculation determines how many participants are needed for the planned analysis of primary outcome to be informative

Calculated using the expected difference/effect size of primary outcome

Question 35

What is the relationship between odds ratio and effect size of primary outcome?

Answer 35

Closer the odds ratio is to 1, smaller the effect size of primary outcome

Question 36

What is the relationship between p values and alpha significance level in determining whether analysis of primary outcome is statistically significant?

Answer 36

If p < alpha (usually 0.05), analysis of primary outcome is statistically significant

Question 37

What is the relationship between confidence intervals and the line of difference in determining whether analysis of primary outcome is statistically significant?

Answer 37

Line of difference is usually 1

If CI crosses line of difference/1, than there is insufficient evidence to suggest that there is statistical significance

If CI doesn’t cross line of difference/1, than there is sufficient evidence to suggest that there is statistical significance

Question 38

What is the relationship between the odds ratio and estimate precision of whether the analysis of primary outcome is statistically significant?

Answer 38

The closer the odds ratio is to 1, the more precise the estimate

Eg. Imprecise estimate: with 95% certainty the true population odds ratio is between 0.69 – 1.36 (between 31% reduction and 36% increase in odds)

Question 39

What 5 factors are needed for sample size calculation?

Answer 39

expected difference for primary outcome

expected baseline events, (usually proportion/rate)

alpha significance level

power (1 - beta) that is usually set at 80-90%

dropout rate

Question 40

What does external validity determine?

Answer 40

Extent to which research evidence can be generalised to other settings

Eg. Are RCT results likely to be applicable to your patient group

Question 41

Give 3 examples of characteristics to compare between local patient population and trial population to determine if there is external validity?

Answer 41

Demographics, clinical features and outcome baseline risk

Question 42

What is the relationship between the eligibility criteria of a RCT and generalisability of the study results, and where in the paper can the eligibility criteria be found?

Answer 42

Inclusion/exclusion criteria (usually found in introduction, method): larger the eligibility criteria for enrolling in the RCT, the better the generalisability of results

Question 43

What part of the paper shows the outcome baseline risk?

Answer 43

Event rate of control group

Question 44

What 5 factors should be assessed when deciding if estimated effect of primary outcome is clinically relevant/significant as well as statistically significant?

Answer 44

minimal difference which is of clinical importance

risks

side effects

administration

costs

Question 45

Give 4 examples of critical appraisal tools?

Answer 45

Cochrane risk of bias tool
CASP checklist
SIGN
NICE