Final Exam Deck

Question 1

How do objective versus subjective endpoints (outcomes) influence the reliability and validity of a RCT?

Answer 1

The RCT becomes more reliable and valid as the endpoints move from subjective to objective.

Question 2

How does one distinguish whether the designated outcomes of a RCT are objective or subjective?

Answer 2

Objective outcomes - Are defined in explicit terms that leave no ambiguity. There is limited personnel for assessing outcomes, and explicit procedures for dealing with conflict.

Subjective outcomes - Surveys and scales are used. Validity is established by assessing whether the outcomes are met, and reliability is established by whether the assessors have been properly trained and are limited in number. MCID is established.

Question 3

Why is it important to establish a minimal clinically important difference (MCID)?

Answer 3

MCID is the minimal amount of difference that is important to patients. Statistically the difference may be meaningless, but to patients the difference is important. There are limitations, however. The consensus of the difference is not determined by patients, the assessment used to determine the MCID may be subject to recall bias, and the change is statistically determined.

Question 4

Would we typically identify the MCID in studies with objective or subjective endpoints?

Answer 4

Subjective endpoints determined through scales and surveys.

Question 5

Is it possible for the results of a study demonstrate statistical significance but not be clinically important? Both statistically AND clinically significant? Neither? Show how by using a diagram which includes the MCID, point estimate & CI. Be sure to point out the direction of benefit and harm with respect to the line of no difference.

Answer 5

Statistical significance w/o clinical significance: When the confidence interval does not cross the null value, but the MCID is greater than the CI.

Statistical significance w/ clinical significance: When the confidence interval does not cross the null value, and the MCID value is lower than the CI (on the left, or w/i the CI on the leftish side).

Neither statistical significance nor clinical significance: The CI crosses the null value, and the MCID value is outside of the CI to the right.

Question 6

What are the advantages and disadvantages of using surrogate endpoints in a RCT?

Answer 6

A surrogate endpoint is used in place of a clinically meaningful endpoint.

Advantages: Fewer patients, shorter trial/faster to market, easier to measure, less invasive for patients.

Disadvantages: Trial is too short to evaluate longterm effects, intervention may effect surrogate endpoint but not clinical outcome, and patient may have a hard time relating the surrogate to the clinical outcome.

Question 7

What are key questions must be addressed when evaluating composite outcomes in a RCT?

Answer 7

Are the components of the outcomes related physiologically so that intervention affects them similarly?
Are the components related such that they are of equal importance to patients?
Is the incidence of the components similar?
Is the treatment effect similar for all components?

Question 8

Compare and contrast P-values and confidence intervals in terms of their helpfulness in establishing RCT reliability.

Answer 8

P-values indicate precision in the abstract, but does not provide good variability or information about clinical significance. Less desirable than CI’s.

CI’s also provide information about precision in rejecting the null, and about variability. There is some information about clinical significance, and CI’s are more desirable.

Question 9

Give an example of how results from a RCT might be statistically non-significant but trend toward a clinically meaningful benefit. What type of error might this scenario represent?
What about statistically significant but not clinically significant?

Answer 9

1. This might indicate a Type II error. The results may show that the intervention will not provide a statistically significant outcome for the patient, but the patient may feel that the intervention works and so the clinical effect of having the intervention is meaningful to the patient. This will have a narrow confidence interval.
2. This might indicate a Type I error. The results from a RCT may be statistically significant in that the confidence interval does not cross the null, but the outcome may not be clinically meaningful for patients. For example, a new antibiotic comes out that statistically will help the patient, but the difference is so minimal that the patient does not feel better.

Question 10

In a non-inferiority trial such as “ESTABLISH-1” what is the purpose of delta? (Δ)

Answer 10

Delta is the clinically acceptable difference. In the ESTABLISH-1 trial (a non-inferiority trial), having a confidence interval w/i the delta means that the treatment is non-inferior or equivalent.

Question 11

For a non-inferiority trial, designate the CI, delta, and ARD on a graph, and interpret the finding.

Answer 11

CI - The range that shows outcome significance.
Delta - The value that is the clinically acceptable difference
ARD - (Absolute risk difference) The range from -delta to +delta that allows the treatment to be non-inferior, equivalent, inferior, or superior.

For a non-equivalency trial, the treatment is non-inferior as long as it does not cross the (-) delta. It may cross the (+) delta to be superior, and that still counts as non-inferior. This is different from an equivalency trial, which requires that the CI be w/i both the (-) and (+) delta.

Question 12

How is the null hypothesis in an Equivalence/Non-Inferiority RCT different than in a Superiority RCT design?

Answer 12

In a superiority trial, the null hypothesis means that there is no difference between treatments. In an equivalent/non-inferioirity trial, the null hypothesis means that there is a difference between treatments.

Question 13

What are some of the issues with multiplicity in a RCT?

Answer 13

Each time there is a new comparison, it increases the chance of a Type I error (where you think there is a difference between groups when there really isn’t any)

Question 14

What are some important considerations when evaluating primary and secondary outcomes in RCTs?

Answer 14

Primary outcomes: Should be clinically relevant. Endpoints should be identified before the analysis.
Secondary outcomes: Should be related and support primary outcomes. Usually involve safety. Explore rather than conclude.

Question 15

When is statistical adjustment needed for multiple outcomes or multiple arms within a RCT?

Answer 15

It is rarely needed for well-designed studies.
It is used when there is more than 1 primary outcome a priori.
It is used when there is a post-hoc change of primary outcome.
It is used when secondary outcomes are being seen as conclusive.

Question 16

What are some considerations for the appropriate planning, analysis, and reporting of subgroup data to preserve the reliability of a RCT?

Answer 16

• Should be pre-specified (identified a priori)
• Clinically justified
• Confined to primary outcome and limited number of subgroups
• Should report all analyses done, not just significant ones.
• Should use statistical tests of interaction to test for effect modification rather than individual p-values
• Rarely should subgroup analysis effect the trial’s conclusions.

Question 17

Define interim analysis and what should reviewers look for if it is present in a RCT?

Answer 17

An interim analysis is an analysis done in the midst of a trial by an independent committee at pre-specified check points and stop the trial for safety or futility.

Be cautious of these!
Multiple interim analyses contribute to multiplicity.

Look out for:
Inadequate power size
Stopping early indicated by incomplete trial size.
A statement that no interim analyses were done (rare)
Trial ended early without pre-specification

Question 18

Define the various types of bias: selection, performance, attrition, detection, reporting

Answer 18

Selection bias- When the selection of the treatment groups was not not done equally.
Performance bias - When the treatment groups are treated differently during the course of their treatment.
Attrition bias - When the patients drop out due to different reasons. This results in the analysis groups being unequal.
Detection bias - Difference in how the outcomes were assessed. This can happen if the outcomes aren’t determined a prior or if those analyzing are not blinded.
Reporting bias - When the participants knowingly falsify data based on a stigma.

Question 19

Explain some benefits and limitations of using RCT assessment tools in terms of the 4 study evaluation domains.

Answer 19

There are four study domains: Validity, Reliability, Applicability (or external validity), and Clinical Importance. RCT assessment tools are focused on internal validity only.

Question 20

Does randomization minimize or eliminate selection bias? Why?

Answer 20

It minimizes bias when the potential confounders are distributed equally between treatment groups. There is a need for allocation concealment and valid randomization procedures. There can still be some bias if the randomization is imbalanced, and this can be minimized again with a large n value.

Question 21

What trial size is needed to limit chance imbalances?

Answer 21

At least an n of 200. (Both simple and restricted randomization)

Question 22

Give some examples of non-random allocation.

Answer 22

If the randomization occurs by a characteristic that a patient has (such as birthdate, age, appearance, date of test result) the groups are less likely to be random.

Question 23

Explain some good allocation concealment methods.

Answer 23

• Use central allocation (IVR, web-based, pharmacy-controlled randomization)
• Use sequentially numbered, opaque, sealed envelopes
• Use sequentially numbered drugs that are identical in appearance

Question 24

Compare and contrast allocation concealment and blinding.

Answer 24

Allocation concealment: Occurs before trial, always achievable, leads to selection bias if not performed
Blinding: Occurs during trial, not always achievable, leads to performance and/or detection bias if not performed.

Question 25

Is blinding always possible? Explain.

Answer 25

No. There are unique side effects profiles of some drugs that are difficult to hide, and some surgeries are also difficult to side. Some ways around this could be sham surgeries, where only the patients are blinded (one-sided blindness)

Question 26

What are some issues with unblinded assessment of outcomes and how can these problems be overcome?

Answer 26

Unblinded assessments of outcomes are prone to performance bias, where the physician treats the patient according to their outcome.
These can be overcome by :
- Single blind
- Double-dummy
- A priori criteria with established set training

Question 27

Why does Intent-to-treat help control for attrition bias?

Answer 27

It preserves randomization, and assumes that people drop out for a reason and not randomly. It represents treatment effect in real world conditions. Assumes that the patients did not have the effect in superiority trials. (Most conservative estimate of treatment effect in superiority trials)

Question 28

Discuss the differences between Intent-to-treat, Per Protocol, “Modified” ITT, and “As treated” analysis.

Answer 28

ITT- Want patients to be analyzed in original groups, and for all patients to be analyzed. Inclusive.
Per protocol - Includes only patients that had optimal adherence to protocol. Not real-life. Loses randomization. Appropriate for safety analyses and non-inferiority trials.
MITT- Excludes patients for reasons not justified.
As treated - Loses randomization because they analyze patients according to group.

Question 29

Demonstrate how to calculate and interpret overall and differential attrition rates for a RCT.

Answer 29

Overall attrition: Add up all the people (all treatment groups) who dropped out of trial. Divide by total number of people that were initially randomized. Rate should be less than 20% to have little attrition bias. (Or greater than 10% between groups)
Differential attrition: The attrition rates of both treatment groups should be calculated and then compared. They should be similar, and the reasons for leaving the trials should be similar.

Question 30

What is a Kaplan-Meir Survival Curve used to demonstrate and what precautions must the RCT reviewer take for proper interpretation?

Answer 30

It uses follow-up data to estimate (conditionally) how a person will survive over a period of time based on specific time intervals. Reviewer must take into account that some patients may be lost to follow-up, that an event doesn’t always occur before the end of the study period, or a patient has a different event which makes follow-up difficult.

Question 31

Describe how “Run-in periods” and other enrichment strategies may affect the external validity of a RTC.

Answer 31

Run-in periods may be used to find a patient population that will have better results. They may be more compliant, or clinician discretion may select certain kinds of patients that will cause an impact on the trial. High rates of exclusion here would be a red flag.