Critical appraisal Flashcards
Randomisation
Purpose is to try and ensure that any characteristics of the sample population that may affect the results ( confounders) are distributed equally between the two study groups, and avoiding selection bias
Tools for randomisation:
centralized computer randomization (with contact by phone or computer) is ideal and often used in multicentre trials.
smaller trials may use an independent person (eg the hospital pharmacy to manage the randomization
Stratified randomisation
With powerful confounders ( eg age, sex) , patients can first be split, or stratified , into different groups before randomisation, so there will be the same number of patients with and without the confounder ( young and old, male and female) in each arm of the trial
allocation concealment
whether the person recruiting the patient to the trial could know or anticipate the group allocation that patient would receive, preventing selection bias
Prevents clinicians predicting which group patients would be in before recruiting them to the trial
Blinding
When some of the partiticapits of a trial ( patient/clinicians/researchers) are prevented from knowing certain information that may lead to conscious or subconscious bias
Blinding advantages
- Prevents observer bias = form of reactivity in which researcher’s cognitive bias causes them to subconsciously influence the participants of experiment = could influence extra quality of care to these patients
- Or confirmation bias = see results that aren’t there
- Expectation bias (Pygmalion effect) = Observers may subconsciously measure or report data in a way that favours the expected study outcome
Also prevents placebo effect – or reduces it
double blinded trial
both patients and investigators are unaware of treatment allocation
when blinding is best to use
outcome is subjective (eg measurement of symptoms or function)
or if outcome measurement is based on patient self report
instances when blinding is not needed
if outcome is objective eg death
sometimes impossible to achieve eg in a trial involving physiotherapy , they will know whether or not they have received it
in a trial involving warfarin ,clinical cannot be blinded due to safety reasons
Confounding
distortion (or potential for distortion) of association between outcome and exposure
by third factor
which has an association with both exposure and outcome
common causes of confounding.
Confounding occurs when there is a non random distribution of risk factors in the populations. Age, sex and social class are common causes
How to control confounding factors
In the design stage of an experiment, confounding can be controlled by randomisation which aims to produce an even amount of potential risk factors in two populations.
In the analysis stage of an experiment, confounding can be controlled for by stratification.
Study least subject to bias
RCT - the groups are likely to be similar with respect to known and unknown determinants of outcome therefore we can be more confident that any observed differences in outcome are due to the intervention.
‘intention to treat’ analysis
Statistical analysis of data from subjects according to the group to which they were assigned despite noncompliance with the study protocol
‘per protocol’ analysis
An analysis of patient outcomes based only on those subjects who completed all aspects of the protocol. Also called on-treatment analysis.
Treatment fidelity
how accurately the intervention is reproduced from a protocol or model
Validity
describes how accurately a study, instrument, test or equivalent measures what it is supposed to.
Factors affecting validity
study size
inter-participant variability the use of different measuring instruments (instrumentality)
Certain biases such as attrition and selection bias
internal validity
how well the study was conducted, the degree to which the effects observed in an experiment are due to the independent variable and not confounds-true, accuracy
Threats to internal validity
Reliability of measurement instruments Regression towards the mean Sampling Experimental mortality Instrument obtrusiveness Maturation Measurement instrument learning
external validity
extent to which we can generalise findings to real-world settings-useful, generalisability
Threats to external validity:
Representativeness of the sample
Reactive effects of setting (is the research setting artificial)
Effect of testing (if a pre-test was used in the study that will not be used in the real world this may affect outcomes)
Multiple treatment inference (this refers to study’s in which subject receive more than one treatment, the effects of multiple treatments may interact)
Reliability
is the extent to which an experiment, test, or any measuring procedure yields the same result on repeated trials.The higher the reliability the more likely you are to obtain similar results if the study was repeated.Reliability does not ensure accuracy.
Cohort study
sample that has been exposed to a certain exposure and follow that sample to observe the outcome. Cohort studies can be retrospective or prospective. used for prognosis and studying rare exposures. But if uncommon event cohort study would have to be unfeasibly large to answer the study question
Cohort study advantages
- Best information about causation of a disease, can work out incidence
- Able to examine a range of outcomes from a particular exposure
- good for rare exposures
Cohort studies negatives
- Often large, difficult to follow up large groups of patients, especially with something such as monitoring diet, expensive and time-consuming
- Hard to conduct if length of time from exposure to outcome is very long (eg for some cancers) or if exposure you’re observing is rare
- Need to look out for confounders
- Bad for long latent periods
- bad for rare outcomes
- Misclassified exposure
- Different follow up for exposed/ non exposed
- Outcomes assessors not blinded to exposure category
- Selection bias eg sample patients all live near a nuclear power plant
Loss of follow up can result in :
- measurement error
reduce available sample size and effect study’s ability to detect a true association between exposure and outcome->increases chance of type 2 statistical error
if it is systematic, it will introduce bias
Limiting confounding in a cohort study
Restriction - limit participants of study that have possible confounders
Matching and stratification - make comparison groups,adjust for confounding
Multiple variable regression - coefficients are established for the confounder groups. Allows for better adjustment
Bradford hill’s criteria
- Strength of association
- Specificity - Does A always only cause B?
- Temporal association - effect has to come after cause
- Theoretical plausibility
- Consistency - Do you always find the same relationship?
- Coherence - Does the data fit in with what we know now?
- Dose-response relationship - Does greater exposure lead to greater effect?
- Experimental evidence - Can we test this experimentally?
- Analogy - If A causes B, does something similar to A cause something similar to B?
Case control study
sample that already has a certain outcome, follow them back to find out if they were exposed to a certain exposure.Case-control studies are retrospective. They clearly define two groups at the start: one with the outcome/disease and one without the outcome/disease. They look back to assess whether there is a statistically significant difference in the rates of exposure to a defined risk factor between the groups.used for Studying cause of rare diseases
Challenges to measuring exposures in case controls:
Recall bias
Variable exposure- patients environment may have changed eg moved house
Unavailable data - eg patient can’t remember, medical records unavailable, incomplete or inaccurate
recall bias
participant cannot remember when they were exposed, or their outcome changes their perception of the exposure
Case-control studies negatives
- Often affected by recall bias participant cannot remember when they were exposed, or their outcome changes their perception of the exposure
- Or affected by selection bias where control group has other factors that may influence their exposure
- Needs a large sample size for rare exposures
- Cases don’t represent the full disease spectrum
- Confounders need to be recognised/addressed
Case control advantages
- simple/easy to conduct, do not require long follow up, outcome already present
- Good for rare outcomes, can select all patients with a certain disease
- Good for long latent periods, not waiting for it long after the exposure
Observational study bias
In an observational study patient or clinician preference rather than randomisation determines whether a patient is allocated to the intervention or comparison group. In the absence of randomisation there is a greater risk of imbalance in both the known and unknown determinants of outcome, and consequently any observed differences in outcome might be unrelated to the intervention but due to differences between groups at baseline
systemtic reviews
A rigorous summary of all the research evidence that relates to a specific question
Finding all the evidence in a systematic review:
Searching bibliographic databases eg medline
Searching for non english papers( avoids language bias)
Searching through references of other trials
Trial registries
Contacting authors
Conducting a thorough search for evidence in a systematic review can eliminate publication bias, language bias and selection bias
Systematic review- how to appraise validity
Is this a systematic review of RCTs? - anything less than RCT is inadequate
What was the search strategy? Studies with negative results or foreign languages are unlikely to be included
How was validity of individual studies assessed?
Are the results consistent from study to study?
disadvantages of single studies:
Individual studies may have inadequate sample sizes to detect important differences (leading to false negative results).The results of apparently similar studies may vary because of chance. Subtle differences in the design of studies and the participants may lead to different or even discrepant findings).
A systematic review has some advantages over a primary data study:
first, speed (why conduct a new RCT if the answer is already available in published research?); second, ethics (is it defensible to conduct a controlled trial if the answer is available in trials already conducted?); third, statistical power (all else being equal, combining data from more than one study increases certainty in findings and precision of estimates).
By bringing together all the relevant evidence, disadvantages of single studies can be guarded against
Sensitivity
Proportion of patients with the disease who get a positive
= No. True positives / all those with disease
Tests with high sensitivity correctly classify a high proportion of people who really have disease
Specificity
Proportion of patients without the disease who get a negative test
= No. Of true negatives / all those without disease
Tests with high specificity correctly classify a high proportion of people who don’t have disease
Positive predictive value
Chance of having disease if your test is positive
No. Of true positives/ all those that test positive
As prevalence Rises this also rises
Negative predictive value
Chance of not having disease if your test is negative
No. Of true negatives / all those test negative
As prevalence increases this value falls
Likelihood ratio for positive test result
•Sensitivity ÷ (1 – Specificity)
Likelihood ratio for negative test result
(1 – Sensitivity) ÷ Specificity
Risk
a ratio of the number of people who develop an outcome to the total number of people
Risk ratio
probability of disease/ risk in exposed/ probability of disease/risk in unexposed
RR<1
means that the treatment decreases risk of the outcome, the rate of an event is decreased compared to controls. The relative risk reduction should therefore be calculated
RR>1
means that the treatment increased the risk of the outcome. the rate of an event (eg experiencing significant pain relief) is increased compared to controls. It is therefore appropriate to calculate the relative risk increase if necessary
RR 1
means no difference between the groups (treatment has no effect)
Number needed to treat (NNT)
number needed to be treated to produce one improved outcome (1 ÷ ARR)
Round up to a whole number
Number needed to harm
number needed to be treated to produce one harmful outcome
Round value down to a whole number
can be calculated by dividing 1 by the absolute risk increase
95% confidence interval
the range of values of the study sample within which we can be 95% sure the true population value lies
statistically significant test result
(P ≤ 0.05)
means that the test hypothesis is false or should be rejected.
Type 1 error
(False positive) Rejecting null hypothesis when it is true shows a difference when there is none Poor study Data manipulation
Type 2 error
(False negative)
Accepting null hypothesis when you should have rejected it/the null hypothesis is false
A type two error occurs when a study fails to detect an effect or an association that does exist. i.e. detects no difference when there is one
Might be related to sample size
Power
probability of correctly rejecting the null hypothesis when it is false
= 1 - probability of type 2 error
the chance of detecting a true difference when there really is one
Power can be increased by
increasing sample size, using more precise measuring instruments, and using a higher significance value.
Odds
a ratio of the number of people who develop an outcome to the number of people who don’t
usually used in case control studies
odds ratio
Odds ratios are always bigger so they look better in a paper abstract. Odds can be >1 but risk is always <1
Multivariate analysis
allow confounding factors to be taken into account, by adjusting for these factors.performed using statistical models.
Hazard ratio
measure of an effect of an intervention on an outcome of interest over time.
Precision
quantifies a test’s ability to produce the same measurements with repeated tests.
Bias
systematic introduction of error into a study that can distort the results in a non-random way
is the tendency of a statistic to overestimate or underestimate a parameter.
Selection bias
Error in assigning individuals to groups leading to differences that may influence the outcome. The subjects are not representative of the population
Especially a problem in cohort studies
Types of selection bias
- sampling bias ( eg due to non random sample of population)
- volunteer/non responder bias
- Time interval bias
- Attrition/ loss to follow up bias
- prevalence/incidence bias (Neyman bias)- study is investigating a condition that is characterised by early fatalities or silent cases
- admission bias (Berkson’s bias)
- healthy worker effect
Recall bias
Difference in the accuracy of recollection of study participants, possible due to whether they have the outcome or not
Especially a problem in case control studies
Publication bias
Failure to publish/include results from valid studies, often because they show a negative or uninteresting result
Important in meta-analyses and systematic reviews where studies showing negative results may be excluded
Hawthorne effect
Group changing its behaviour due to the knowledge that it is being studied
Procedure bias
Subjects in different groups receive different treatment, other than just the interventions
Eg elderly patients could receive human contact along with intervention whereas control patients may get no extra human contact
Measurement bias
It occurs when the accuracy of information collected about or from study participants is not equal between cases and controls
Quantitive eg: due to poor calibration of measuring instruments
Qualitative eg: study participant is less likely to answer a question due to stigma associated with the answer
Lead time bias
Early diagnosis appears to prolong survival
Length bias
Screening over represents less aggressive disease
A reason why cancers detected by screening may on average be more slowly progressive.
screening tends to detect disease which progresses more slowly and has long asymptomatic periods.. Conversely, aggressive disease is more likely to be missed by screening
Effective Reproduction Number
is the average number of secondary infections produced by a typical infective agent; if this number is greater than 1 then it is impossible to eradicate an infection.
Cost Effectiveness Analysis (CEA):
Used when the effect (outcome) of the two interventions is expected to vary
The outcome is measured in natural units e.g. BP, cholesterol level, mortality, live years saved
The outcome is one dimensional - addresses quantity or quality, not both
Costs & outcomes are combined into a single measure to allow comparison
Cost Utility Analysis (CUA):
Used when the effect (outcome) of the interventions on health status has two or more dimensions
Measures outcome in terms of quantity and quality. Combines these into a single measure e.g. the QALY = Quality Adjusted Life Year
A measure which tries to combine a quantitative measure (months gained, years gained) with a qualitative measure of the quality of that measure
Can be used to compare interventions with a disease or condition or across different diseases or treatment options
QALY
Based on number of years of life that would be added by intervention
Each year of perfect health = 1, to a value of 0 for death
If extra years are not lived in full health (eg loss of limb) = between 0 and 1
Cost Benefit Analysis (CBA):
Places a monetary value on benefits or outcomes
Generally based on individuals’ observed or stated preferences and values for something
Most common approach is “willingness to pay”
most comprehensive but rarely undertaken.
Cost Minimisation Analysis (CMA):
Used when the effect (outcome) of both interventions is identical (or assumed to be identical)
No outcome measurement
Only costs are accounted for
cannot provide answers where effectiveness is different between competing alternatives
Cost Consequence:
Measures costs, measures consequences
Sensitivity Analysis:
‘Vary key assumptions and see if it has an impact’
Incremental cost effectiveness ratio
to work out cost effectiveness
This is the ratio of the change in costs of therapeutic interventions (compared with alternatives/no treatment) to the change in effects of the intervention ( measured as clinical outcome of QALY)
ICER=(Cost A- Cost B) / (QALYs b -QALYsA)
Why - Combining data in a meta analysis
Increases the number of patients being analysed ( increases size of study sample)
Improves precision and reduces the width of the confidence intervals
Can demonstrate a statistically significant result when none of the trials could do this individually
(only poss with homogenous data eg mortality)
Regression to the mean
People often get better or worse regardless of intervention = not the intervention causing change
Equipoise:
Equipoise refers to the situation where the researchers have no preference between the treatments being studied in a trial
When it is lost: when those designing the trial or recruiting patients into it do have a preference for one treatment over another, it can be considered unethical to recruit patients into the trial as you are offering some patients what you believe to be a “worse option”
