Week 8. Evaluating Diagnostic Literature

Question 1

Validity

Answer 1

• Is it true? Can I believe it? Are the outcome measures trustworthy and accurate?
• Extent that a measure assess what it is intended to measure.

Question 2

Applicability

Answer 2

If valid and important, can/should I apply it to my patients

Question 3

Use of Diagnostic Tests

Answer 3

PT’s have increased access to DI but it should not replace clinical assessment/tests
E.g. Shoulder imaging — physicians order shoulder imaging to facilitate referral to surgeon but after prolonged wait periods, surgeon refers patient to PT

Question 4

Diagnosis Research Goals

Answer 4

1. Evaluate whether a test gives additional information about presence/absence of a condition
2. Evaluate whether clinical test provides similar information as an invasive or radiological test
3. Evaluate whether a diagnostic test is able to distinguish between patients with and w/o a specific condition
4. Avoid invasive tests/x-ray exposure, more carefully define injured structures/tissues to customize treatment

Question 5

Clinical Prediction Rules (CPR)

Ottawa Ankle Rules

Answer 5

• Used for diagnosis
• A rule or model that tries to identify the best combination of S&S, and other findings for predicting the probability of a specific outcome

OAR
- sensitivity: 96-99% 
- specificity: 26-48% 
if negative, low chance of fracture 
(point tenderness at posterior edge (of distal 6 cm) or tip lateral malleolus. point tenderness at posterior edge (of distal 6 cm) or tip medial malleolus. inability to weight bear (four steps) immediately)

Question 6

Level of evidence in diagnostic study design:

Why can’t RCT be used in Dx studies

Answer 6

• RCT cannot be done in Dx studies as all subjects must undergo both tests
  Level 1 evidence in Dx studies
• Cross-sectional
• Cohort study designs

Question 7

Methodological Issues in Diagnostic Research

• Gold Standard Test

Answer 7

Inappropriate gold standard/reference test

Question 8

Methodological Issues in Diagnostic Research

• Verification Bias

Answer 8

Verification Bias:

results in test influence the decision to have the gold standard test

Question 9

Methodological Issues in Diagnostic Research

• Selection/referral bias

Answer 9

Selection/Referral Bias:

Evaluation done in a Population with a high prevalence of disease or investigators pick study participants

Question 10

Methodological Issues in Diagnostic Research

• Measurement Bias

Answer 10

Measurement Bias

• Testers are aware of gold standard tests results which bias outcome
• Outcomes for what constitutes positive/negative are not well-defined
• Testers unable to complete diagnostic test properly

Question 11

Sensitivity (SnNout)

Answer 11

Sensitivity: likelihood of a +test in presence of disease (true positive rate)

SnNout:

• a negative result on a highly sensitive test is a good way to rule out people who don’t have the condition
• Example: airport security, if highly sensitive will pick up all kinds of metal, so no buzz = no metal, but lots of false positives, but you don’t miss things

Question 12

Specificity (SpPin)

Answer 12

Specificity: likelihood of a -test in the absence of disease (true negative rate)

SpPin: a highly specific test will not falsely identify people has a condition, a positive result on a highly specific test is likely to accurately detect the presence of a condition
- Example: airport security, if airport sensor is turned down, it would be highly specific (metal = buzz)

Question 13

Positive Prediction Value

Answer 13

Likelihood of disease in the presence of +test

Question 14

Negative Predictive Values

Answer 14

Likelihood of not having a disease in the presence of a negative test

Question 15

Positive/Negative predictive values table

• Rows calculate?
• Columns calculate?

Answer 15

Rows = Predictive Values
Columns = sensitivity/specificity 

(TP)True+ (FP) False+ Total who test positive
(FN)False- (TN) True+ Total who test negative

Total w. Total w/o
Disease. Disease

Question 16

Accuracy

Answer 16

Accuracy = (a + d) / (a+b+c+d)

= (TP + TN) / (TP + TN + FP + FN)

Question 17

Sensitivity calculation (true positive rate, TPR)

Answer 17

Sensitivity = a / (a+c)

• true positive divided by total number with disease.
• this is the probability of positive test if subject has disease, also called true positive rate

Question 18

Specificity Rate (True negative rate, TNR)

Answer 18

Specificity = d/ (b+d)

• computed as true negatives divided by total number without disease
• defined as probability of negative test if subject does not have disease; true negative rate (TNR)

Question 19

Positive Predictive Value

Answer 19

PPV = a / (a+b)

• computed as true positive divided by total number that tested positive
• defined as probability of disease if subject has a positive test; true negative rate (TNR)

Question 20

Negative Predictive Value

Answer 20

NPV: d / (c+d)

• true negative divided by total number that tested negative
• probability of no disease if subject has a negative test

Question 22

Specificity vs. NPV vs. -LR

Answer 22

Specificity (d / b+d)

  - do not have disease 
  - probability of negative test 

NPV (d / c+d)

  - negative test 
  - probability of no disease

-LR: probability of a negative test result given the presence of the disease and the probability of a negative test result given the absence of the disease, i.e.

Question 23

Which aspect is dependent on prevalence of disease?

Answer 23

PV are dependent on prevalence of disease, while sensitivity/specificity are not

• PV are meaningless out of context of prevalence
• Sensitivity and Specificity are dependence on diagnostic threshold; more consistent BETWEEN studies

Question 24

Sensitivity and specificity

- more reliable INTER- or INTRA?

Answer 24

• most consistent between studies

- diagnostic threshold for a sp diagnostic test defined as min or max requirement to obtain a positive result

Question 25

Receiver Operator Characteristic Curves (ROC Curves)

Answer 25

• 3-way relationship between sensitivity, specificity, and diagnostic threshold
• curve shows trade-off between sensitivity and specificity with changing diagnostic thresholds

ROC values
0 = terrible
1 = ideal

Question 26

Positive Likelihood Ratio - ratio indicates? - probability that a person? - Value indicates?

Answer 26

Sensitivity / (1-specificity) - ratio of true positive rate to false positive rate - probability of a person with a positive test has the disease - larger numbers indicate higher likelihood of disease

Question 27

Negative LIkelihood Ratio - ratio indicates? - probability that a person? - Value indicates?

Answer 27

NLR: (1 - sensitivity) / specificity - ratio of true negative rate to the false negative rate - used to determine the probability with a negative test does not have the disease - smaller numbers indicate higher likelihood of NO disease

Question 28

LR+ of 7.29 and LR- of 0.166 If individual takes test for the disease, we can update their probability of disease by multiplying odds by the likelihood ratio

Answer 28

If test is positive, updated odds of disease: (1/99) x 7.29 = 0.0736 If test is negative, updated odds of disease (1/99) x 0.166 – 0.00168 Odds of disease increases from 1% to 7.4% with positive test and decreases from 1% to 0.16% with a negative test

Question 29

``` +LR. -LR. > 10. < 0.1 5-10. 0.1 - 0.33 3-5. 0.34 - 0.99 <3. > 1 ```

Answer 29

Almost conclusive Useful Marginally Useful Likely not important

Question 30

Clinical Utility of DI Statistics: Sensitivity/Specificity

Answer 30

- most common reported values | - can calculate LR from these values

Question 31

Clinical Utility of DI Statistics: PV

Answer 31

- limited useful ness because less stable estimates (depends on population tested/prevalence of disease)

Question 32

Clinical Utility of DI Statistics: LR

Answer 32

- Most clinically useful because they contain both sensitivity/specificity values in 1 ratio

Question 33

Example: A new ‘special test’ for the shoulder has been developed to test for a presence of a rotator cuff tear Want to compare results of the new test to a known standard ``` A = 50 B = 10 C = 15 D = 25 ``` ``` Accuracy? Sensitivity? Specificity? PPV? NPV? +LR? -LR? ```

Answer 33

``` Accuracy = 75/100 Sensitivity = 77% Specificity = 71% PPV = 83% NPV = 62% +LR = 2.7 (likely not important) -LR = .32 (may be useful) ```

Question 34

Sensitivity vs. PPV vs. LR+

Answer 34

Sensitivity (a / a+c): probability of a positive test if subject has disease (TPR) - they have the disease - probability of positive test recognizing PPV (a / (a+b): probability of disease if subject has a positive test - they have a positive test - probability of actually having disease LR+ (sensitivity / (1-specificity)): probability of a positive test result given the presence of the disease and the probability of a positive test result given the absence of the disease,