evaluating study design (MUST MEMORISE) Flashcards
evaluating studies
🧠 Memorisable Evaluation Sentences by Study Design
📊 Ecological study
“While ecological studies are useful for identifying population-level trends and generating hypotheses, they are limited by the ecological fallacy, where associations at the group level may not reflect individual-level risk.”
📝 Cross-sectional study
“Cross-sectional studies offer a snapshot of exposure and outcome at a single time point, making them efficient and low-cost, but they cannot establish temporality and are prone to reverse causation.”
🟨 Cross-sectional studies:
✅ Good for: Estimating prevalence, generating hypotheses, identifying associations.
❌ Limited because:
They measure exposure and outcome at the same time → you can’t tell which came first.
No follow-up = no idea of cause-effect direction.
📌 So yes, they are weaker for causal inference because they can’t establish temporality.
Memorisable sentence:
“As a descriptive cross-sectional study, it is useful for identifying associations but cannot infer causality due to lack of temporality.”
🟩 Ecological studies:
✅ Good for: Quick, population-level comparisons; hypothesis generation.
❌ Limited because:
Use group-level data, not individual data.
Risk of ecological fallacy (assuming group trends apply to individuals).
📌 Also weaker for causal inference, especially compared to case-control or cohort studies.
Memorisable sentence:
“Ecological studies are limited by the ecological fallacy and confounding, making them the weakest design for establishing causal relationships.”
🧾 Case-control study
“Case-control studies are efficient for rare outcomes and allow evaluation of multiple exposures, but they are vulnerable to recall and selection biases, and cannot directly estimate incidence or prove causality.”
🧍♂️🧍♀️ Cohort study (prospective or retrospective)
“Cohort studies can establish temporality and assess multiple outcomes, but they may require large sample sizes, long follow-up, and can be affected by loss to follow-up and confounding.”
🧪 Randomised Controlled Trial (RCT)
“RCTs are the gold standard for causal inference due to randomisation reducing confounding, but they may have limited generalisability due to strict inclusion criteria and ethical/practical constraints
evaluating source populations + specific examples?
- Representativeness
Ask: Does this population reflect the target population (or real-world patients)?
🧠 Memorable sentence:
“Although the study sample is large, its representativeness may be limited if the source population differs from the wider population in key characteristics such as age, comorbidities, or healthcare access.”
🔹 2. Recruitment / Sampling Bias
Ask: How were people included? Could certain groups be over- or under-represented?
🧠 Memorable sentence:
“Selection bias may arise from how participants were sampled, particularly if inclusion depended on factors related to the exposure or outcome, such as access to hospital care or likelihood of diagnosis.”
🔹 3. Restrictions / Exclusions
Ask: Were there criteria that made the population too narrow?
🧠 Memorable sentence:
“Restrictive eligibility criteria, such as excluding older adults or patients with comorbidities, may limit generalisability and reduce external validity.”
⚡ Bonus: Tailored Sentences for Common Source Populations
📦 Database (e.g., BIFAP, UK Biobank)
“Large databases offer valuable real-world data, but lack of control over inclusion and variable data quality can introduce bias and affect the reliability of exposure or outcome measures.”
Defined Population:
What: Biobank participants are often from a specific region, time, or group (e.g., volunteers, hospital patients).
Implication: Might not match the broader population you care about (e.g., all PTC patients).
High-Quality Data (Sometimes):
What: Biobanks often have standardized, detailed records (e.g., tumor samples, ultrasound reports).
Implication: Can reduce measurement error, but only if the data fits the study’s needs.
🏥 Single-centre hospital
“Single-centre designs allow for data consistency but may limit generalisability, as patient populations and clinical practices may not reflect those in other regions or healthcare settings.”
but it may also introduce selection bias, as the patients may come from a limited geographic area or share similar characteristics that could limit the generalizability of the findings.
“Moreover, since the study was conducted at a single centre, there may be a bias toward including more severe cases of PTC, as patients with milder forms of the disease may seek treatment at different facilities. This could skew the results, potentially leading to an overestimation of recurrence rates, as more severe cases are often associated with higher recurrence.”
🌍 Multicentre / National registry
“Multicentre or national registry studies improve generalisability and diversity but may vary in data completeness and quality across sites.”
🔁 How to Practise:
Use this formula to structure your evaluation:
“The study population was drawn from [source]. While this provides [strength], it may limit [limitation], particularly in relation to [representativeness/recruitment/restrictions].”
introducing misclassification and how internal validity effects external validity
“Differential misclassification could occur if stroke cases’ supplement use is recalled or recorded more accurately due to medical scrutiny, while nondifferential misclassification might arise from general prescription errors affecting all groups equally.”
“These methodological limitations, including biases and incomplete adjustment, challenge the study’s internal validity and its generalizability to broader populations at risk of ischemic stroke.”
why its good to adjust for confounders
“Adjustment for key confounders, such as lymph node metastasis and tumor size, strengthens the causal interpretation of the observed associations.”
segway into the implications of a design critiques?
Use transition phrases to connect critique to implications:
“Nonetheless, these design limitations must be considered when interpreting the study’s conclusions.”
a better design?
Reinforce critical engagement: Consider adding a sentence that reflects on how the study design could be improved or what a better alternative might be, e.g.:
“A multi-center prospective design could have mitigated selection bias and improved representativeness.”
“While biobanks offer extensive data, recruitment is often voluntary and self-selecting; therefore, population-based sampling strategies could have reduced selection bias and improved generalisability.”
📂 Routine clinical databases (e.g., BIFAP, CPRD)
“Although clinical databases are valuable for real-world evidence, linkage with broader national datasets or prospective data collection could enhance control over exposure and outcome measurement.”
registries or administrative data:
“Use of registry data improves sample size and generalisability, but prospective cohort designs could offer greater control over confounders and more accurate outcome classification.”
what to say about a large sample
“The study’s large sample size and long follow-up period provide substantial statistical power to detect associations between location and recurrence.”
the impact of bias
“These methodological choices and biases raise questions about the study’s ability to generalize findings to diverse PTC populations.”
evaluating any observational study
“While observational studies are valuable for hypothesis generation, they cannot establish causal relationships, as confounding, selection bias, and reverse causation may distort the observed associations.”
bringing in the bradford hill criteria
Memorisable Sentence to Bring in the Bradford Hill Criteria:
“The study suggests a causal relationship, but based on the Bradford Hill criteria, several factors such as temporality and biological plausibility need further examination to support this conclusion fully.”
“Although the study identifies a strong association, it lacks sufficient evidence in terms of consistency and biological gradient, which are key for establishing causality according to the Bradford Hill criteria.”
when medical / researcher people are interpreting stuff
Blinded review: Were the interpretations blinded to the study’s outcomes (e.g., recurrence rates) to minimize bias in how the ultrasounds were interpreted?
Quick Rule: Suggest blinding if subjective interpretation (e.g., scans, diagnoses) by pros/volunteers could skew results—put it in discussion as a fix, not methods (they didn’t do it).
