Lecture 12 - Hierarchy of Evidence

Question 1

Hierarchy of evidence

Answer 1

• Ranking system (levels) of various study designs
• What study design provides the most robust evidence
– That is most free of bias and hence most rigorous
– The higher up in the hierarchy, the more likely the study can minimise the impact of bias
– You can then choose the best evidence to answer your question

Question 2

Why do we need a ranking system?

Answer 2

• The research design of the study can influence the believability of the results of the study
– Internal validity
• Relates to the conduct of the research
• Can you be confident that the research was well conducted so that the effect you notice is from the exposure or intervention (causal relationship)
– External validity
• Generalisability of the research findings from the sample to the reference population

Question 3

What can affect internal validity?

Answer 3

• Chance (imprecision) (minimised through strategies)
– Random error
– Cannot be eliminated but minimised
– Usually through having sufficient sample size
• Bias
– Systematic error
– Errors in the way research was undertaken – Can be eliminated
• Confounders
– Variables that were not taken into account and
hence influence outcome

Question 4

Threats to internal validity

Selection/sampling bias

Answer 4

• Bias (systematic error)
– Selection/Sampling bias
• Who gets selected for the study and who gets left out
• Sample should represent the wider population
– Addressed through clear and methodologically sound sampling
strategies (eg: Random selection)

Question 5

Threats to internal validity

Allocation bias

Answer 5

– Allocation bias
• Who gets into the intervention group and who gets into the control group
– Addressed through random allocation

Question 6

Threats to internal validity

Maturation bias

Answer 6

– Maturation bias
• Changes which occur naturally over time
– Addressed through having a “true” control group

Question 7

Threats to internal validity

Attrition bias

Answer 7

– Attrition bias
• Drop outs from the study
• Changes the sample which can then affect the results
– Adequate sample size to begin with (with loss to follow
already taken into account)
– Sample size calculations before the start of the research

Question 8

Threats to internal validity

Measurement bias

Answer 8

– Measurement bias
• Errors in measuring data/variables
– Outcomes from an intervention
• ? Expectations of the researchers/health professionals
– Training and achieving standardisation in the data collection process
– Blinded measurer/assessor

Question 9

Threats to internal validity

Placebo

Answer 9

– Placebo
• Participants expect or believe that the intervention will result in an improvement
– Participant blinding
» However, not always possible in allied health

Question 10

Threats to internal validity

Hawthorne effect

Answer 10

– Hawthorne effect
• Participants experience changes due to the attention given to them as part of the research
– Pleasing the researcher
» Commonly seen in clinical practice
» Participant blinding

Question 11

Threats to internal validity

Confounding

Answer 11

• Confounding
– Meaning to confuse
– Can causally influence the outcome
– Do not cause error in measurement but rather error in interpretation
– Random allocation spreads confounders randomly and evenly
• Unknown confounders are an issue
– Eg. Motivation to participate in the intervention

Question 12

How to choose hierarchy of evidence ?

Answer 12

• There is no one gold standard hierarchy of evidence
• There are many hierarchy of evidence available around the world
• So, which one do you choose?
Dependent on your question 
• Levels of Evidence – Intervention
– Diagnosis
– Prognosis
– Aetiology
– Screening intervention

Question 13

NH&MRC Hierarchy
In Australia
Intervention

Answer 13

New NH&MRC Hierarchy 2009
Intervention
Level - Intervention
I - A systematic review of level II studies
II - A randomised controlled trial
III-1 - A pseudorandomised controlled trial
(i.e. alternate allocation or some other method)
III-2 - A comparative study with concurrent controls:
- Non-randomised, experimental trial
- Cohort study
- Case-control study
- Interrupted time series with a control group
III-3 - A comparative study without concurrent controls: Historical control study
-Two or more single arm study ‡
-Interrupted time series without a parallel control group
IV - Case series with either post-test or pre-test/post- test outcomes

Question 14

NH&MRC Hierarchy
In Australia
Diagnosis studies

Answer 14

I - A systematic review of level II studies
II - A study of test accuracy with: an independent, blinded comparison
with a valid reference standard, among consecutive patients with a defined clinical presentation
III-1 A study of test accuracy with: an independent, blinded comparison with a valid reference standard, among non-consecutive patients with a defined clinical presentation
III-2 A comparison with reference standard that does not meet the criteria required for Level II and III-1 evidence
III-3Diagnosticcase-controlstudy
IV - Study of diagnostic yield (no reference standard)

Question 15

NH&MRC Hierarchy
In Australia
Prognostic Studies

Answer 15

• I - A systematic review of level II studies
• II - A prospective cohort study
• III-1 All or none of the people with the risk factor(s) experience the
outcome
• III-2 Analysis of prognostic factors amongst untreated control patients in a randomised controlled trial
• III-3 A retrospective cohort study
• IV - Case series, or cohort study of patients at different stages of disease

Question 16

NH&MRC Hierarchy
In Australia
Aetlogical* Studies

Answer 16

• I - A systematic review of level II studies
• II - A prospective cohort study
• III-1 All or none of the people with the risk factor(s) experience the
outcome
• III-2 A retrospective cohort study
• III-3 A case-control study
• IV - A cross-sectional study
• * Where intervention based studies can’t be trialled due to ethical issues

Question 17

NH&MRC Hierarchy
In Australia
Screening intervention*

Answer 17

• I - A systematic review of level II studies
• II - A randomised controlled trial
• III-1 A pseudorandomised controlled trial (alternate allocation or some
other method)
• III-2 A comparative study with concurrent controls:
– ▪ Non-randomised, experimental trial
– ▪ Cohort study
– ▪ Case-control study
• III-3 A comparative study without concurrent controls:
– ▪ Historical control study
– ▪ Two or more single arm study
• IV – Case series
• * Screening interventions are designed to identify disease in a community early, thus enabling earlier intervention and management in the hope to reduce mortality and suffering from a disease. Eg: In asymptomatic adult women at risk of breast cancer (population), does screening (intervention) reduce likelihood of breast cancer fatality (outcome) compared with routine self- examination (comparator)?”

Question 18

Deficiencies on NH&MRC Hierarchy

Answer 18

• The NH&MRC hierarchy do NOT recognize opinions of respected authorities as evidence levels
– Editorials, opinion pieces, points of view – Consensus statements
• If there is an absence of published evidence then there is NO evidence

Question 19

Hierarchy of Evidence
Overseas approaches
Lloyd-Smith hierarchy

Answer 19

1a Meta-analysis of randomised controlled trials
1b One individual randomised controlled study
2a One well-designed, non-randomised controlled study
2b Well-designed quasi-experimental study
3 Non-experimental descriptive studies – comparative/case studies
4 Respectable opinion

Question 20

Hierarchy of Evidence
Overseas approaches
Harbour and Miller hierarchy

Answer 20

1++ High quality with low risk of bias (MA or SR 1+ Well conducted with a low risk of bias (as
RCT’s) above)
1- High risk of bias (as above)
2++ High quality sys rev of case controls or
cohort studies with very low risk of confounding, bias and a high
probability that the relationship is causal
2+ Well conducted case control or cohort s with a low risk
of bias (as above)
2- Case control or cohort studies with a high risk of bias
(as above)
3 Non-analytic studies
4 Expert opinion

Question 21

Hierarchy of Evidence
Overseas approaches
Oxford Centre for Evidence-based Medicine Levels of Evidence (2011)

Answer 21

• Diagnosis
• Prognosis
• Treatment Benefits
• Treatment Harms
• Screening

Table in lec notes

Question 22

Hierarchy of Evidence
Overseas approaches
Scottish guideline group (SIGN)

Answer 22

1a Meta-analysis of RCTs
1b At least one RCT
IIa At least one well designed controlled study without randomization
IIb At least one other type of well designed quasi- experimental study
III At least one well-designed non-experimental descriptive study, such as comparative studies, correlation studies and case studies
IV Expert committee reports or opinions, and/or clinical experiences of respected authorities

Question 23

Hierarchy of Evidence
Overseas approaches
Alternative evidence-rating scale (Bleck 2000)

Answer 23

• Level 0 - Things I believe
• Level 0a - Things I believe despite the available data
• Level 1 - RCT’s that agree with what I believe
• Level 2 - Other prospectively collected data
• Level 3 - Expert opinion
• Level 4 - RCT’s that don’t agree with what I believe
• Level 5 - What you believe that I don’t

Question 24

Qualtitative research

Answer 24

• Hierarchy of evidence generally not applied in qualitative research
– Why?
• Recent attempts
– Generalizable studies (level I)
• Provides detailed overview about a topic with sound rigour and trustworthiness
– Conceptual studies (level II)
• Provides some in depth focus on a particular issue and highlights varying viewpoints with some limitations
– Descriptive studies (level III)
• Provides a broad overview about a particular issue and hence superficial
– Single case
• Focus on a single group and hence not transferrable
• Ongoing debate on its use