Critical appraisal Flashcards
What is meany by the Number Needed to Treat (NNT)
The NNT measures the number of patients who need to receive a treatment to prevent one additional bad outcome
equation for NNT
In studies it is calculated by 1 / (EER - CER)
i.e. NNT = 1 / ARR
what is meant by relative risk
RR = Probability of an event when exposed/Probability of event in control group
Also called risk ratio.
measure of proportionate increase in exposed group
RR = EER / CER (= (a / a+c) / (b/b+d) )
what is meant by absolute risk?
Absolute risk is likelihood of developing a disease over a time period
what is meant by hazard ratio
A hazard ratio considers your absolute risk to be 1.
If something you do / take doesn’t change your risk, then the hazard ratio is 1.
If something you do lowers your risk by 30% compared to not doing it then that action makes your hazard ratio 0.70 = the risk is 70% of what it was (30% lower).
If something you do triples your risk, then your hazard ratio is 3.0 (risk is 3 times greater than it was before).
Experimental event rate calculation
EER = a/ (a+c)
a = exposed and diseased c = exposed and not diseased
A measure of how often a particular event occurs within the experimental group of a trial.
what is the Absolute risk reduction
ARR (RD) = CER - EER
i.e. (a/a+c) / (b/b+d)
absolute risk reduction is the same as risk difference. Comes from subtracting the risk in one group from the risk in the other group.
High risk patients show a bigger risk difference than low risk patients. Patients who are unlikely to experience the outcome will need more patients to be treated for one outcome to be avoided, and the risk difference is smaller.
what does it mean if the relative risk is 1
no risk difference
no causation
what does it mean if the relative risk is > 1
Increased risk amongst those exposed
i.e. it is a risk factor
what does it mean if the relative risk is < 1
decreased risk amongst those exposed
i.e. it is a protective factor
what is relative risk reduction
Relative risk reduction is the proportional reduction in risk between active and control participants in a trial.
If the relative risk reduction is 10%, the risk in the active treatment group will be 90% of that in the control group.
what makes the NNT larger?
Less effective treatments
i.e. prevent fewer events, so more patients need to be treated to prevent a single event.
what makes NNT smaller?
The NNT gets smaller for longer durations of treatment.
Or for more effective treatments
Which will appear more impressive of the relative risk or the absolute risk?
The relative risk will always be a larger percentage figure than absolute risk, so relative risk seems more impressive.
What impact does the framing of research results have
The framing of results is the way that results are presented and this can influences doctors’ attitudes about a treatment.
what is EER
EER = experimental event rate (incidence in group receiving intervention) EER = a/ (a+b)
what is CER
CER = control event rate (incidence measured outcome in control) CER = c / c+d
What is meant by odds ratio
Odds Ratio is a method of comparing how likely events are between 2 groups.
odds of event occurring in one group divided by the odds of event occurring in the other group.
OR = a x d / b x c
calculation for odds of a positive outcome in exposed patients
odds f a positive outcome in exposed patients = a/b
e.g. odds of rolling a 1 on a dice = 1/5 = 0.2
what is a Student’s t-Test
Students T-test compares two means (averages) and tells you if they are different from each other.
And how significant the differences are.
what is Chi Squared test
Chi Squared test is a
Nonparametric test used to compare numerical or categorical data sets.
what is categorical variable
ategorical variables represent types of data which may be divided into groups.
Examples of categorical variables are race, sex, age group, and educational level.
what is meant by Nonparametric data
Does not fit a normal distribution.
Nonparametric statistics uses data that is often ordinal, meaning it does not rely on numbers, but rather on a ranking or order of sorts.
e.g. a survey conveying consumer preferences ranging from like to dislike would be considered ordinal data.
what is a numerical / quantitative (Non-categorical) variable
Numerical (Non-categorical) variables are usually referred to as range variables and are expressed as numbers within the set of the Reals, usually varying from negative infinity to positive infinity
what statistical tests can be done with data that is normally distributed
Students T test
ANOVA (analysis of variance)
what statistical tests can be done with data that is not normally distributed
nonparametric tests - which dont assume anything about the population parameters.
chi-square,
Fisher’s exact test
Mann-Whitney test.
what is a regression analysis used for
A regression equation is used in stats to find out what relationship, if any, exists between sets of data
Regression analysis is also used to understand which among the independent variables are related to the dependent variable.
In restricted circumstances a regression analysis can be used to infer causal relationships between the independent and dependent variables.
what does simple linear regression result in
A linear relationship between one dependent variable and one independent variables = a straight line of best fit
Like an average of where all the points line up.
In linear regression, the regression line is a perfectly straight line
what does polynomial regression result in
A polynomial regression results in a curved line of best fit .
polynomial regression is considered to be a special case of multiple linear regression
what does multiple linear regression result in
Also known as multiple regression
A statistical technique to model the linear relationship between the 2 pr exploratory (independent) variables and the response (dependent) variable.
linear relationship between one dependent variable 2 or more independent variables.
What are Rank Sum tests?
Wilcoxan and Mann Whitney U tests are both rank sum tests.
These are Nonparametric tests
Does not require the assumption of normal distributions.
What are Bradford Hills criteria and what are they for?
The criteria for causation
- Temporality - time sequence
- plausability
- consistency
- strength of association
- dose - response gradient
- reversibility
- specificity
- coherence
- analogy
points to look at when appraising a case control study?
- focused aim / issue
- Is the method appropriate for the question
- Recruitment - case definition, cases representative of population, system for case selection, are cases incident or prevalent, time frame of stuy relevant to disease?, power calculation
- contol selection - representative and matched?
- Exposure accurately measured? recall bias, subjective, is the measure validated
- blinding
- confounders
- results
- strenght of association, OR, P-value, CI
what is a consort flow diagram
a flow diagram to provide a broad overview of how the trial was conducted
Concise method to report research method and protocol
Shows progress through phases of an RCT of each trial arm
Advantage of case control studies
cheap Quicker than cohort requires smaller sample size not prone to loss to follow up study rare outcomes
Define positive predictive value
how likely someone who tests positve actually has the disease
= true positive
PPV = A/A+B
(where A is screen +ve has disease and B is screen positive no disease)
Define negative predictive value
How likely someone who tests negative actually does not have the diease
= true negative
PPV = D/C+D
(where D is screen -ve no disease and C is screen -ve has disease)
what is the attributable risk
Same as the risk difference.
difference between incidence rates in the exposed and non exposed groups
calculation for control event rate (CER)
CER = b / b + d
b = not exposed and diseased c = not exposed and not diseased
calculation for estimated population exposure
a + c / a + b + c + d
calculation for (population attributable risk percent) PAR%
PAR% is calculated by
dividing the population attributable risk (PAR) by the incidence in the total population
then multiplying the product by 100 to obtain a percentage
How is relative risk different to attributable risk?
Relative risk helps determine whether and how strongly a precursor is associated with a particular outcome.
Attributable risk helps determine how much of an
outcome may be attributable to a particular risk factor (i.e. an estimate of the excess risk) in a
population exposed to that factor.
what is a population attributable risk?
The proportion of all cases of an
outcome in the total population that could be attributed to the exposure to the risk factor.
what factors help distinguish cause from association?
strength of association (relative risk) time sequence - cause MUST precede outcome distribution - geographical gradient - dose and duration correlation with outcome consistency specificity biologically plausable experimentally reproducable precentative trials
what is the absolute risk?
Number of events (or disease) / total number in the group or population
uses of descriptive studies
guidance on association
demonstrate a broad geographical difference
determine relative frequency of dieases or factors
service planning
evaluating service effectiveness
types of sampling methods for observational studies
simple random sampling - e.g. random number generator
systematic sampling - seclect person at regular intervals
stratified sample - propabbility of patient selection varies by a pre-determined characteristic to ensure a subgroup is represented
cluster sample - groups are the sampling units, randomly selected
continuous sampling
convenience sampling
sources of bias in sampling
deviation from sampling methodology
omission of people who are hard to identify as belonging to the population
large numbers declined to participate
self selection or volunteer patients
excessive exclusion criteria
Berkson bias - selection taken from a sub-population who do not reflect the whole population (e.g hospital patients)
What does CONSORT stand for?
Consolidated Standards Of Reporting Trials
what is CONSORT used for
Consolidated Standards Of Reporting Trials
A statement introducing a set of recommendations to improve RCT reporting
Protocol to identify problems from conducting RCTs and assist with reporting them fully and clearly
what are the 4 stages of an RCT listed in a CONSORT flow diagram?
- Enrollment
- Intervention allocation
- Follow - up
- Analysis of data / results
The first part of a CONSORT flow diagram is ‘enrollment’
what 3 things should this include?
Enrollment - include
1 - Number assessed for eligibility
2 - Number excluded and why - not meeting inclusion criteria / decline / other reasons
3 - Number randomised (study population)
The second part of a CONSORT flow diagram is ‘allocation’
what should this include?
Allocation includes
- number of patients allocated to intervention
- number of patients in each arm who receive the intervention
- number in each arm who do not receive the intervention
The third part of a CONSORT flow diagram is ‘follow-up’
what should this include?
Follow-up should include - for each arm
- number lost to follow up
- reasons for loss to follow up
- number discontinuing the intervention
- reasons for discontinuing
The fourth part of a CONSORT flow diagram is ‘analysis’
what should this include?
Analysis includes - for each arm
- number analysed
- number excluded from analysis
- reasons for exclusion from analysis
Disadvantages of a retrospective case control study
Retrospective recall bias + reduced reliability incomplete records previous records not standardised Difficult to prove causality Difficult to select appropriate controls Cannot calculate incidence within a population
Why is an intention to treat analysis usually done?
What is the benefit of this?
subjects are analysed according to the treatment allocation irrespective of whether is was recieved
This is because treatment allocation was random.
Any changes to treatment were not random.
ITT avoids breaking randomisatin
Therefore not doing an ITT analysis introduces a potential for confounding.
ITT analysis demonstrates the potential effect of the treatment protocol rather than the specific treatment.
What is a ‘rate’ in relation to studies
The relative frequency of an event per unit of time
The calendar time period must be the same for the numerator and denominator
often expressed as per 1000 per year
What is PICO
Population / patients
Intervention
Control / comparator
Outcome
In PICO what should be considered in relation to ‘P’
Population / patients
- age
- gender
- ethnicity
- diagnosis
In PICO what should be considered in relation to ‘I’
Intervention
- the treatment or exposure
In PICO what should be considered in relation to ‘C’
Control / comparison
- control / non exposed
- placebo
- current best treatment
In PICO what should be considered in relation to ‘O’
Outcome - must be measurable
- pain score
- survival time
- cure / remission
- shorter hospital stay
Points to look at when appraising an RCT
Aims and primary outcome - clarity Recruitment and selection Randomisation Allocation concealment Controls and matching Confounders Exclusion criteria Blinding Drop out rates - why sample size - power calculation results - p-values, CI ITT analysis or per-protocol
what is the risk of an under powered study
A small sample size leads to an under-powered study.
This increases the risk of making a type 1 error
= where it is believed that a differnce exists where no such difference does exist
What is a type 1 error
where researchers believe an effect is demonstrated when really there is no effect
I.e to incorrectly reject the null hypothesis
Methods of correcting for multiple statistical analyses
Bonferroni - P-value / number of tests
Benjamini-Hochberg procedure - involves ranked p-values from smallest to largest and allocating the smallest rank to one, rank 2 to the next and so on. Then compare each to the B-H critical value
Sources of error
- Subject Variation = differences in the observed factor
- e.g. variation in BP on different occasions
- impact of motivation, recall etc
- Observer variation
- intra-observer variation
- inter-observer variation
- bias due to awareness of the hypothesis
- inconsistent procedure or question wording
- lack of skill / training / interest
- Technical Limitations
- Test doesnt measure what it is intended to
- method intrinsically unreliable
- defective instruments
- incorrect calibration
- test repetition and replication
Advantages of prospective cohort studies
Prospective = better able to standardise methods and reduce error
Can calculate incidence in exposed and non-exposed groups
Can distinguish cause from association
Can study several outcomes from one hazard
disadvantages of prospective cohort studies
Long duration
Large number of subjects
Expensive
May have higher drop out rate / loss to follow up
cannot be certain of true causation - this requires experimentation
Why is it usually preferable for a study to recruit incident rather than prevalent cases?
Incident cases are newly diagnosed and have less risk of confounders
Prevalent cases are patients with already established disease - therefore they represent the survivors of an earlier cohort of incident cases.
Prevelent cases are less likely to represent everyone who is diagnosed as it doesn’t include those who get better or die.
management of confounding variables in a case-control study
matching of controls
exclusion criteria
subset analysis based on confounding variables
multivariate analysis technique e.g. logistic regression
Calculation of Rate %
(number of cases / population at risk) x 100
Hierarchy of evidence
Meta analysis / Systematic review RCT Non randomised CT Cohort study Case control study Cross sectional survey Case series Case report Editorials, expert oppinions Animal research
Define sensitivity
the ability of the test to correctly identify those patients with the disease.
= patient does have the disease and is positive on the test
A/ a+c
define specificity
the ability of the test to correctly identify those patients without the disease.
= patient does not have the disease and is negative on the test
What does the positive predictive value depend upon?
PPV and NPV are dependent on the population being tested
Are influenced by the prevalence of the disease
Define likelihood ratio
How much more likely is it that a patient who tests positive has the disease compared with one who tests negative.
likelihood that a given test result would be expected in a patient with the target disorder compared to the likelihood that that same result would be expected in a patient without the target disorder.
5 steps of EBM
1) form a question re a clinical problem
2) evidence
3) critical appraisal
4) application
5) implementation and monitoring
Items to consider if appraising a website
Is there a named author? Author qualifications? Funding / supporting of site listed? Any conflicts of interest recorded? Is the site hosted by a reputable organisation - e.g. .org / .gov / .edu Any peer review process Is the coverage full and unbiased does the spelling, grammar and information appear accurate?
What does the term ‘MeSH search’ mean?
Medical Subject Headings search - uses pre-defined keywords
Types of observational descriptive studies
Survey / questionaire
Qualitative survey
Case series
Case report
Types of observational analytical studies
Case control study
Cohort study
Types of experimental studies
Randomised control trial Controlled trial Uncontrolled trial Cross over trial N of 1 trial
What is an ecological study
A study on a population or community
What is a pragmatic trial?
Takes place in a real life environment therefore may be more reflective of everyday practce
what is an economic analysis
Assess cost and/or utilities of intervention
Helps to prioritize services
What is a systematic review
A group of studies which are grouped and appraised together to answer a hypothesis
What is a meta-analysis
A statistical procedure for combining the numerical data of separate `studies to answer a clinical question
What is internal validity
To what extent does the study measure what it set out to measure.
What is the evidence that the intervention caused the outcome
Did the study limit and control confounding variables
Robust research design
Sufficient power
Correct statistical analysis
Factors affecting internal validity
Attrition / mortality (drop outs) maturation - respondants grow out of the condition / grow experinenced instrumentation regression towards the mean selection contamination history per-protocol analysis
What is external validity
The extent to which results from the study can be generalised to the wider population
Can the results be applied to real patients in real life settings
Can the results be generalised to other locations and times
Factors affecting external validity
Sample population representative of wider population?
Sampling method
Does the test / measurement affect behavior or outcome
does the test environment mirror real life
is the test stimuli similar to real life
do the elicited behaviours represent real life behaviours
Define efficacy vs effectiveness
Efficacy = the impact of interventions under optimal trial conditions
Effectiveness = the impact of the interventions under clinical or real life circumstances
Considerations regarding the journal when appraising evidence
Is it peer reviewed - independently, non-editorial staff
Is peer review double-blind?
Impact factor - based on number of citations
Disadvantages of the peer review system?
delay in publication
author may be known to the reviewer - therefore reduces objectivity
revolutionary / unpopular opinions may face opposition within peer review
doesnt guarantee errors will not appear
How is a journal impact factor calculated?
Ranking of journals by measuring the frequency of the average article cited per year
Done by reuters
Comparison of journal performance
What is the Indelfinger rule?
Neither original research
or any of its pictures or tables
may be published in more than one outlet
Types of conflict of interest
Factors which may influence professional judgements
Financial - grants / honoraria
Professional - membership to an organisation
Personal
criteria of authorship
Any of the following
- Substantial contribution to conception
- or design of the work
- or acquisition / analysis / interpretation of data for the work
Risk of multiple subgroup analysis
‘data dredging’
Increased risk of Type 1 Error
more analyses = more likely that a significant result result will be found by chance
- unless the p-value is corrected
Risk associated with adding subgroup analyses after the data is collected
Introduces bias
increases risk of making a type 1 error
research method and power may be sub-optimal to answer additional questions
Difference between bias and confounding
Bias leads to wrong results - the data is incorrect
Confounding leads to the wrong conclusion - but the results are correct
What causes bias
Mistakes in the research techniques
The results generated are therefore wrong
Bias is not a real-life association or relationship but is introduced by the research process
What causes confounding?
A real-life relationship that exists between exposure and outcome independent of the research process.
NOT a mistake
Studies have the correct result but the wrong conclusion may be reached if confounders are not recognized
2 common types of bias
Selection bias
Observation bias
What is selection bias
Problem with recruitment and allocation
Sample not representative of target population
includes - sampling bias and response bias
What is observation bias
Problem with the way data is gathered
Results influenced by expectations of research team or subjects
Types of selection bias
Sampling bias Response bias Berkson / admission rate bias Diagnostic purity bias Neyman bias / incidence-prevalence bias Membership bias Historical control bias
What is a Berkson Bias
Admission rate bias
e.g. sample taken from inpatient setting but the hospital cases do not represent the severity of the condition in the general population
What is a diagnostic purity bias?
Excessive exclusion criteria
Comorbidity is excluded from the sample to such an extent that the sample doesnt reflect the complexities of the general population
What is a Neyman bias?
Incidence-prevalance bias
Occurs if prevalence of a condition doesnt reflect its incidence
Often affects rapidly progresssive conditions
Gap between onset of the condition and selection of sample means some individuals with the condition are no longer available due to death
Therefore sample = milder form - but generalised to whole population
What is a membership bias
When membership to a group is used to recruit subjects - members may not be representative of the population due to motivation etc
What is historical control bias
Subjects and controls are selected across time
Changes in definition, exposures, diseases and treatment can mean that the subjects and controls cannot validly be compared
What is a response bias?
type of selection bias
Individuals volunteer for the study but differ in some way from the general population
e.g volunteers are motivated
can arise from advertising study e.g. radio / papers
Types of observation bias
Interviewer / ascertainment bias
recall bias
response bias
hawthorne effect
What is Interviewer / ascertainment bias
Researcher is not blinded to the subjects status in the trial which affects their approach to recording results
What is recall bias
Subjects selevtively recall details from the past
Esp in case control studies or surveys
What is response bias
Can occur in any study where the patient is asked questions
Patients may answer in the way they think the researcher wants to hear - rather than their true feelings ./ beliefs
what is the Hawthorne effect
Subjects alter their behaviour when they are aware they they are being observed in a study
What is performance bias?
Systematic differences exist between groups in the care provided
or in exposure to factors other than the intervention of interest
Reduced by
- blinding
- randomization
- standardization of care protocols
What is attrition bias
When the numbers of individuals dropping out of the studies differ significantly between the study arms
Incomplete outcome data
Those at the end may not be representative of those who started and the target population
What is a qualitative survey?
Gathering opinions of a group of people.
Emphasis on subjective meaning and experience
Used to study complex issues
What is a case report
The experience of a single person described
Usually anecdotal
Usually cannot be replicated
Can be used to generate a hypothesis
What is a case series?
The experiences of a group of people is described
Useful for studying rare diseases
Can you critically appraise a case study or case series?
Not really
But consider
- Did the study address a clearly focused question / issue?
- Is the researcher’s perspective described / taken into account?
- Is the patients perspective described / taken into account?
- Are the conclusions drawn justified by the case?
- Are the findings of the study relevant to other settings?
What is an observational analytical study
Reports similarities and differences observed between 2 or more groups
- case control study
- cohort study
What is a case control study
Type of observational analytical study
Retrospective
People who have the outcome variable are compared to people who do not - to determine risk factors they have been exposed to in the past
Advantages of a case control study
Cheap
Easier than other studies
few subjects required
suitable for studying rare outcomes / those with a long time lag between exposure and outcome
Disadvantages of a case control study
Rely on recall or health records = source of bias
Unsuitable for rare exposures as this would require a v large sample
can be difficult to select a control group - need to be similar to the cases group
What is a cohort study
Type of observational analytical study
Prospective
Group of people followed to see how development of n outcomes differs between those with an exposure and those without
Advantages of a cohort study
Useful for rare exposures
can assess multiple outcomes
can assess temporal relationships between risk factors and outcomes
can directly estimate outcome and incidence rates
Disadvantages of a cohort study
May take a long time from exposure to outcome
Expensive
Problems with dropu outs / loss to follow up
unsuitable for rare outcomes as would need a v. large sample
How is a ‘retrospective cohort study’ conducted?
Using a pre-existing cohort
Use a historical group of patients who had an exposure and contact them now to see if they have developed the outcome of interest - avoids need to wait
