Midterm 1: RCT Triage Flashcards
What to look for when assessing participants and generalizability?
- is there a clear description of who was eligible to participate (e.g. inclusion/exclusion criteria)
- is there a table of baseline data giving a detailed description of the participants
- are conclusions aimed at a population similar to the participants
What to look for when assessing primary and secondary outcomes (Outcomes 1)?
- Has a primary outcome been stated (if more than one parameter is measured, the primary outcome should be clearly identified)
- Are the conclusions based on the primary outcome (conclusions about effectiveness that are based on a secondary outcome are not valid)
- red flags are studies with no specific outcome that measure many parameters and are likely to mistake false positives for treatment effects (determining/comparing the effects of x)
What to look for when assessing reporting (Outcomes 2)?
- Are all results given for all outcomes, or are results for any of the stated outcomes missing?
- Are absolute values given and not just percentages?
- Are confidence intervals given and not just significance level?
- Are conclusions based on any within-group comparisons to baseline? (look to see that means were compared to eachother and not to baseline within the same group)
What to look for when assessing sample size?
- Did the authors state that they did a sample size calculation?
- Are the four variables listed (alpha/significance level, power, effect size, variance)?
- Were the analyzed groups at least as large as the sample size or was the trial underpowered?
What to look for when assessing randomization?
- Was the randomization method truly random? (coin flip, random number generator)
- If the sample size < 100, was a restricted method used (stratification or blocking)?
What to look for when assessing allocation concealment?
- Was the person responsible for placing participants into groups aware of what treatment the participants would be receiving?
- Does the paper mention centralization, allocation, third-party (pharmacist, online allocation tool)?
- If anything, did the paper use SNOSE? (this is not as secure but can still be effective if employed properly)
What to look for when assessing blinding?
- Did the authors explicitly state who was blinded? (specific groups should be named; look for participants, treatment providers, etc)
- Were the treatments sufficiently similar (in colour, taste, smell) to prevent participants from knowing which treatment they were assigned? (and was this described in detail)
- If no blinding was performed, was the outcome objective or subjective (fatal flaw if subjective)?
What to look for when assessing participant losses?
- Did the authors provide a flow chart that shows all losses with reasons for losses?
- Did the authors use ITT analysis? (authors may wrongly call PP a modified ITT or simply ITT; look at number of participants in beginning and end of trial and what number they used in the analysis)
- Are losses > 10%
What to look for when assessing interpretation?
- Are the authors conclusions justified by the data in the tables and figures?
- Have the authors compared their findings to existing evidence and if the findings are new, have they given adequate justification?
- Are limitations stated?
What to look for when assessing harms?
- Is there a heading titled “Harms”
- Is there a table listing adverse effects in each group?
- Do the authors balance harms and benefits when drawing conclusions, or do they ignore harms or downplay them?
What to look for when assessing funding?
- What source is the funding (government, industry, or non-profit?)
- Is a for-profit funder involved in the research? (check COI statement; is the company involved in any aspects of the research or manuscript; are authors employees of the company)
- Is there a conflict of interest statement?
- Are there other conflicts of interest? (E.g. has an author received a personal consulting fee from a for-profit funder?)
Why is it important to have a primary outcome and only draw conclusions from the primary outcome?
Since we cannot have an infinitely large group in a trial (to decrease the risk of unevenly distributing confounding variables during randomization), scientists accept a risk level of 5% (α=0.05), meaning that 1/20 trials will show a false positive.
The risk value only applies to one measurement, so the more outcomes that are measured, the higher the chances of observing a difference between groups that is due to heterogeneity between groups and not due to the treatment.
Since the sample size measurement is based on the primary outcome, conclusions about efficacy can only be based on the primary outcome.
Why is it important to report results for all outcomes measured?
Some outcomes require support from other outcomes in order to support the findings in the trial. Cherry picking and reporting only positive outcomes can be misleading, if the positive outcomes are not supported. Only one positive outcome with a lack of support from other outcomes can indicate a high chance of a false positive.
Why is it important to list values as exact values and not percentages?
Listing only percentages can be misleading as it can act to exaggerate the effect of the treatment (ex. drug reduces risk of dying from disease by 50% vs. drug reduces risk of dying from disease from 2/1,000,000 to 1/1,000,000).
Why is it important to give CI and not just alpha (significance level)?
α relies to heavily on the mean and does not take into account the magnitude of the effect observed. Sometimes, α lies in the range of medically important but the confidence interval for the same group may cross over into not medically important, rending the treatment uneffective. Therefore, CI are more informative than just giving α.
