Critical Appraisal Flashcards
Define Validity
Is the finding true
Define Generalisability
Is the finding applicable elsewhere?
3 reasons for not being valid
Chance (random error) —- (finding imprecise) Bias (systematic error) —- (finding inaccurate) Confounding (error of interpretation) —- (beware, unknown confounders cannot be statistically accounted for)
Define
Efficacy
Vs
Effectiveness
Efficacy: Shows if something works under ideal conditions
Vs
Effectiveness: shows if something works under normal conditions
Types of research Qs:
Pragmatic research
Vs
Explanatory research
Pragmatic research - demonstrates effectiveness Vs Explanatory research - demonstrated efficacy
Aspects of
Pragmatic research
Unselected population
Pt centred outcomes
Does not interfere with clinical practice
Aspects of
Explanatory research
Can it work under ideal conditions
Specific staff/setting/population
Often clinical outcome
Null hypothesis
Define
No difference between active treatment and placebo
P value
Define
Chance null hypothesis is true
Type 1 error =
False positive (p value)
Type 2 error =
False negative
Statistical significance =
alpha + beta
Define a+b
Alpha
- maximum possibility of a false positive (p value)
Beta
- max possibility of false negative
(Determined by sample size)
Power of study =
1- beta
Beta = max possibility of a false negative
Usually set to 80-90%
Study power is set by
- Level of alpha
- Sample size
- The minimum clinical difference you wish to detect
Issue with multiple hypothesis testing:
Increases false positive
- only use if there is a clear rationale to the test
Benefit of randomisation
Removes confounders
Concealment prevents
Allocation bias
Blinding prevents
Measurement bias
Outcome bias
More likely if
Soft outcomes
Intention to treat
All analysis
Protects against bias
Loss to follow up
Assume
Patients list are similar to analysed group.
Unless >30% LTFU
Or difference between test and control group
Absolute risk =
Intervention - control
Relative risk =
Intervention-control / proportion
NNT=
1 / absolute risk
Cluster randomisation
Issues
Allocation of concealment not possible
Therefore bias introduced
Hawthorne effect =
Change due to being monitored
Sustainability
What additional resources are required
What negative effects could occur due to intervention being rolled out
Generalisability
X
Independence of reference standard
X
Work up bias
When a different reference standard is used
Incorporation bias =
Diagnostic test forms part of reference standard
Reliability =
Intraobserver error
Effect of prevalence on
Sensitivity and specificity
PPV
NPV
Sensitivity and specificity — remains constant PPV -increases with increasing prevalence NPV - decreases with increasing prevalence
Likelihood ratio =
Value of information added
I =no value
>10 is diagnostic
<0.1 is rule out
Forest plot =
Graph for heterogeneity
In a meta analysis
Pros and cons
Meta analysis
Increased precision but same accuracy
Does not overcome bias of original data
Areas of evaluation
For critical appraisal
Therapy
Service organisation
Diagnostic test
Systematic reviews
Write an abstract
OD PICO RACS
Objective
Design
Population
Intervention
Comparator
Outcome
Results
Adverse events
Conclusions
Studies (further)
Two main questions that critical appraisal tries to answer
1) is the result valid
2) is it generalisable
Probability estimates of random error include
P value
Confidence intervals
Main factor to consider in non randomised data
Confounders
Inaccurate
Vs
Imprecise
Estimates
Inaccurate - due to bias
Vs
Imprecise - random error
Confidence intervals and
Precision
Wide
Vs
Narrow
Wide - imprecise
Vs
Narrow - precise
Statistics used to estimate effect of random error on results
2 types
P value
- for hypothesis testing
Confidence intervals
- for estimation of differences between groups
P values and alpha values
And associations with false/true positive/negative
P value = probability of false positive (type 1 error)
Alpha value = probability of a true positive
(Max probability of false positive)
Beta value = probability of false negative
Allocation or selection bias
How it occurs
And prevention
If patients or researchers can choose which treatment they get
Prevent via randomisation and concealment of allocation
How to randomise service organisation as a intervention
Cluster randomisation
Disadvantage - no concealment if allocation
80/20 rule
As a chart
Pareto chart
Bar chart with
cumulative line as percentage
Likelihood ratio
The Likelihood Ratio (LR) is the likelihood that a given test result would be expected in a patient with the target disorder compared to the likelihood that that same result would be expected in a patient without the target disorder.
