Lecture 12 - Hierarchy of Evidence Flashcards
Hierarchy of evidence
• Ranking system (levels) of various study designs
• What study design provides the most robust evidence
– That is most free of bias and hence most rigorous
– The higher up in the hierarchy, the more likely the study can minimise the impact of bias
– You can then choose the best evidence to answer your question
Why do we need a ranking system?
• The research design of the study can influence the believability of the results of the study
– Internal validity
• Relates to the conduct of the research
• Can you be confident that the research was well conducted so that the effect you notice is from the exposure or intervention (causal relationship)
– External validity
• Generalisability of the research findings from the sample to the reference population
What can affect internal validity?
• Chance (imprecision) (minimised through strategies)
– Random error
– Cannot be eliminated but minimised
– Usually through having sufficient sample size
• Bias
– Systematic error
– Errors in the way research was undertaken – Can be eliminated
• Confounders
– Variables that were not taken into account and
hence influence outcome
Threats to internal validity
Selection/sampling bias
• Bias (systematic error)
– Selection/Sampling bias
• Who gets selected for the study and who gets left out
• Sample should represent the wider population
– Addressed through clear and methodologically sound sampling
strategies (eg: Random selection)
Threats to internal validity
Allocation bias
– Allocation bias
• Who gets into the intervention group and who gets into the control group
– Addressed through random allocation
Threats to internal validity
Maturation bias
– Maturation bias
• Changes which occur naturally over time
– Addressed through having a “true” control group
Threats to internal validity
Attrition bias
– Attrition bias
• Drop outs from the study
• Changes the sample which can then affect the results
– Adequate sample size to begin with (with loss to follow
already taken into account)
– Sample size calculations before the start of the research
Threats to internal validity
Measurement bias
– Measurement bias
• Errors in measuring data/variables
– Outcomes from an intervention
• ? Expectations of the researchers/health professionals
– Training and achieving standardisation in the data collection process
– Blinded measurer/assessor
Threats to internal validity
Placebo
– Placebo
• Participants expect or believe that the intervention will result in an improvement
– Participant blinding
» However, not always possible in allied health
Threats to internal validity
Hawthorne effect
– Hawthorne effect • Participants experience changes due to the attention given to them as part of the research – Pleasing the researcher » Commonly seen in clinical practice » Participant blinding
Threats to internal validity
Confounding
• Confounding
– Meaning to confuse
– Can causally influence the outcome
– Do not cause error in measurement but rather error in interpretation
– Random allocation spreads confounders randomly and evenly
• Unknown confounders are an issue
– Eg. Motivation to participate in the intervention
How to choose hierarchy of evidence ?
• There is no one gold standard hierarchy of evidence • There are many hierarchy of evidence available around the world • So, which one do you choose? Dependent on your question • Levels of Evidence – Intervention – Diagnosis – Prognosis – Aetiology – Screening intervention
NH&MRC Hierarchy
In Australia
Intervention
New NH&MRC Hierarchy 2009
Intervention
Level - Intervention
I - A systematic review of level II studies
II - A randomised controlled trial
III-1 - A pseudorandomised controlled trial
(i.e. alternate allocation or some other method)
III-2 - A comparative study with concurrent controls:
- Non-randomised, experimental trial
- Cohort study
- Case-control study
- Interrupted time series with a control group
III-3 - A comparative study without concurrent controls: Historical control study
-Two or more single arm study ‡
-Interrupted time series without a parallel control group
IV - Case series with either post-test or pre-test/post- test outcomes
NH&MRC Hierarchy
In Australia
Diagnosis studies
I - A systematic review of level II studies
II - A study of test accuracy with: an independent, blinded comparison
with a valid reference standard, among consecutive patients with a defined clinical presentation
III-1 A study of test accuracy with: an independent, blinded comparison with a valid reference standard, among non-consecutive patients with a defined clinical presentation
III-2 A comparison with reference standard that does not meet the criteria required for Level II and III-1 evidence
III-3Diagnosticcase-controlstudy
IV - Study of diagnostic yield (no reference standard)
NH&MRC Hierarchy
In Australia
Prognostic Studies
• I - A systematic review of level II studies
• II - A prospective cohort study
• III-1 All or none of the people with the risk factor(s) experience the
outcome
• III-2 Analysis of prognostic factors amongst untreated control patients in a randomised controlled trial
• III-3 A retrospective cohort study
• IV - Case series, or cohort study of patients at different stages of disease
NH&MRC Hierarchy
In Australia
Aetlogical* Studies
• I - A systematic review of level II studies
• II - A prospective cohort study
• III-1 All or none of the people with the risk factor(s) experience the
outcome
• III-2 A retrospective cohort study
• III-3 A case-control study
• IV - A cross-sectional study
• * Where intervention based studies can’t be trialled due to ethical issues
NH&MRC Hierarchy
In Australia
Screening intervention*
• I - A systematic review of level II studies
• II - A randomised controlled trial
• III-1 A pseudorandomised controlled trial (alternate allocation or some
other method)
• III-2 A comparative study with concurrent controls:
– ▪ Non-randomised, experimental trial
– ▪ Cohort study
– ▪ Case-control study
• III-3 A comparative study without concurrent controls:
– ▪ Historical control study
– ▪ Two or more single arm study
• IV – Case series
• * Screening interventions are designed to identify disease in a community early, thus enabling earlier intervention and management in the hope to reduce mortality and suffering from a disease. Eg: In asymptomatic adult women at risk of breast cancer (population), does screening (intervention) reduce likelihood of breast cancer fatality (outcome) compared with routine self- examination (comparator)?”
Deficiencies on NH&MRC Hierarchy
• The NH&MRC hierarchy do NOT recognize opinions of respected authorities as evidence levels
– Editorials, opinion pieces, points of view – Consensus statements
• If there is an absence of published evidence then there is NO evidence
Hierarchy of Evidence
Overseas approaches
Lloyd-Smith hierarchy
1a Meta-analysis of randomised controlled trials
1b One individual randomised controlled study
2a One well-designed, non-randomised controlled study
2b Well-designed quasi-experimental study
3 Non-experimental descriptive studies – comparative/case studies
4 Respectable opinion
Hierarchy of Evidence
Overseas approaches
Harbour and Miller hierarchy
1++ High quality with low risk of bias (MA or SR 1+ Well conducted with a low risk of bias (as
RCT’s) above)
1- High risk of bias (as above)
2++ High quality sys rev of case controls or
cohort studies with very low risk of confounding, bias and a high
probability that the relationship is causal
2+ Well conducted case control or cohort s with a low risk
of bias (as above)
2- Case control or cohort studies with a high risk of bias
(as above)
3 Non-analytic studies
4 Expert opinion
Hierarchy of Evidence
Overseas approaches
Oxford Centre for Evidence-based Medicine Levels of Evidence (2011)
- Diagnosis
- Prognosis
- Treatment Benefits
- Treatment Harms
- Screening
Table in lec notes
Hierarchy of Evidence
Overseas approaches
Scottish guideline group (SIGN)
1a Meta-analysis of RCTs
1b At least one RCT
IIa At least one well designed controlled study without randomization
IIb At least one other type of well designed quasi- experimental study
III At least one well-designed non-experimental descriptive study, such as comparative studies, correlation studies and case studies
IV Expert committee reports or opinions, and/or clinical experiences of respected authorities
Hierarchy of Evidence
Overseas approaches
Alternative evidence-rating scale (Bleck 2000)
- Level 0 - Things I believe
- Level 0a - Things I believe despite the available data
- Level 1 - RCT’s that agree with what I believe
- Level 2 - Other prospectively collected data
- Level 3 - Expert opinion
- Level 4 - RCT’s that don’t agree with what I believe
- Level 5 - What you believe that I don’t
Qualtitative research
• Hierarchy of evidence generally not applied in qualitative research
– Why?
• Recent attempts
– Generalizable studies (level I)
• Provides detailed overview about a topic with sound rigour and trustworthiness
– Conceptual studies (level II)
• Provides some in depth focus on a particular issue and highlights varying viewpoints with some limitations
– Descriptive studies (level III)
• Provides a broad overview about a particular issue and hence superficial
– Single case
• Focus on a single group and hence not transferrable
• Ongoing debate on its use
