Critical Appraisals Of RCT

Question 1

Why critically appraise?

Answer 1

• If we practice evidence-based medicine….
• Clinicians and policy makers need to determine validity and relevance of research studies
• Consider whether clinical practice/ policy should change as a result

No Research stuffy is perfect

Question 2

What is critically appraisal?

Answer 2

Process of systematically examining research evidence to determine its trustworthiness, value and relevance

In practice it involves answering a series of questions and formulating an overall opinion of quality and relevance of a piece of research

Question 3

Critical appraisal: RCT

Answer 3

Two overall steps:
1. Internal validity
2. External validity

Question 4

Critical appraisal: RCT

Answer 4

Two overall steps:
1. Internal validity – the extent to which causal conclusion can be reached
(can the findings be otherwise explained by chance, bias, confounding?)
2. External validity – the extent to which study findings can be generalised to other settings

Question 5

Internal validity

Answer 5

•The aim of an RCT is to compare groups equivalent in all respects other than the treatment itself
• Randomisation can only produce groups that are comparable at the start of the study
• Other aspects of good trial design are required to retain comparability to the end
• RCTs are nevertheless at risk of bias

Question 6

Internal validity

Answer 6

1. What is the research question and is an RCT the correct methodology to answer it?
2. Was assignment to treatment groups randomised? If so, how was it done? Was treatment allocation concealed? If so, how was it done?
3. How similar were treatment groups at baseline?
4. Was the study blinded? If so, who was blinded?
5. Aside from the allocated treatment, were both groups treated the same throughout the study?
6. Were all patients accounted for at the end of the study? Were all patients analysed according to their originally allocated groups (intention-to-treat)?
7. What was the primary outcome? Is it relevant? Was it pre-specified? Is there evidence of selective reporting of outcomes?
8. Was analysis of the primary outcome statistically significant? How precise was the estimate (95% confidence intervals)? Were the assumptions of the sample size calculation met?

Question 7

External validity (generalizability)

Answer 7

1. Are the results of the trial likely to be applicable to your patient group?
2. Is the estimated effect clinically relevant/significant? (i.e. not just statistically significant)

Question 8

Internal validity

Answer 8

What is the research question?
-Try to frame the question using PICOT - Participants
- Intervention
- Comparator
- Outcome (primary) - Timeframe

Question 9

Is an RCT the correct/best methodology to answer it?

Answer 9

Is an RCT the correct/best methodology to answer it?
•If testing an intervention usually the answer is yes
- the experimental design and comparable groups (active vs control)
- allow causality to be more easily ascribed (compared with observational research)
•Some research questions cannot be answered by RCTs
- What effect does smoking have on the incidence of lung cancer?
- Does a certain genetic trait increase the risk of type 2 Diabetes?
- What motivates women to attend screening for cervical smear tests?

Question 10

Was treatment allocation concealed? If so, how was it done?

Answer 10

-Mechanism which prevents foreknowledge of treatment assignment
-The decision to recruit a patient, and informed consent obtained in ignorance of the next assignment of the sequence
-Allocation concealment prevents selection bias -(not to be confused with blinding!)
-Acceptable methods
- use a third party – pharmacy or central telephone/web based randomisation system - seal drugs in sequentially numbered identical containers according to allocation sequence

Question 11

How similar were treatment groups at baseline?

Answer 11

-Comment on proportions of individuals with reported characteristics (demographic, lifestyle, clinical) between groups
-It is important to ensure the groups are comparable (provides additional assurance that groups only differ with respect to treatment allocation)
-Provides additional evidence that findings are causal (and not explained by selection bias or confounding)

Question 12

Were all patients accounted for at the end of the study? - internal validity

Answer 12

Yes
(the numbers add up)

Question 13

Were all patients analysed according to their originally allocated groups (intention-to-treat)?

Answer 13

Yes
The patients were analysed according to their original allocated groups

Question 14

What was the primary outcome? Is it relevant?

Answer 14

-Should be explicitly reported in the manuscript -Does it help answer the research question?

Question 15

Was the primary outcome pre-specified? Is there evidence of selective reporting of outcomes?

Answer 15

• Best determined by comparing the protocol with the manuscript
• Should be pre-specified
  expected difference/effect size for primary outcome informs the sample size calculation
  prevents selective reporting of outcomes

Question 16

External validity (generalizability)

Are the results of the trial likely to be applicable to your patient group?

Answer 16

-How do the characteristics of your local patient population compare with the trial population? In terms of demography and clinical characteristics
Baseline risk of the outcome
-Look at the inclusion / exclusion criteria
-Look at the baseline table of characteristics -Look at the event rate in the control group -Compare the above with your local population

Question 17

Are the results of the trial likely to be applicable to your patient group?

Answer 17

-Inclusion and exclusion criteria

-Baseline characteristics

-Baseline risk of the outcome

Question 18

Is the estimated effect clinically relevant/significant? (i.e. not just statistically significant)

External validity (generalizability)

Answer 18

•Need to consider minimum difference which is of clinical importance
•Assessment of risks, treatment side effects, administration (simple vs. time consuming and complex)
•Costs