Classification in Diagnostic Tests

Question 1

What is sensitivity?

Answer 1

Sensitivity is the true positive rate of a test or measure. This means the percentage of participants, of who you know that they have a certain disorder/pathology, are correctly classified as having a certain pathology/disorder/etc.

Question 2

What is specifity?

Answer 2

Specifity is the true negative rate of a test or measure. This means the percentage of participants who belong to the healthy control group get also classified as having no disorder/pathology.

Question 3

What is the positive predictive power of a test?

Answer 3

This is the confidence you can have (in terms of percentage) that an outcome of a test is actually predicting what the measure should be predicting.

Question 4

What is negative predictive power?

Answer 4

Related to specifity; the confidence you can have in that an outcome of a test which says that someone has no pathology/disorder is in reality also predicting this.

Question 5

Which of these TOC values are the most important for clinical practice?

Answer 5

PPP and NPP

Question 6

Which factor influences PPP and NPP? And which implication does this have for clinical practice?

Answer 6

Prevalence rates of a given disorder; if this is higher, the PPP will increase. If this is a lower prevalence, the NPP will increase. This means that you really have to be careful to have the same base rate of the disorder in your sample as is the case in the population, otherwise you will have over- or underestimations of PPP and NPP.

Question 7

What is the current problem with TOC values in NP-research?

Answer 7

That most researchers only include calculations of sensitivity and specifity, and don’t consider PPP and NPP, thereby excluding important information. Meaning, that clinicians have (with only S and S) no idea how confident they can be that a certain measure rules in/out a disorder correctly.

Question 8

Clinicians need information on true positives (1), true negatives (2), but also false positives (3) and false negatives (4) when using a test or measure for clinical decision-making.

Answer 8

1. Sensitivity
2. Specifity
3. PPP
4. NPP

Question 9

What are the main recommendations for TOC-values according to Lange & Lippa?

Answer 9

(1) Always calculate sensitivity, specificity, PPP, and NPP.
(2) Calculate PPP and NPP values based on a range of hypothetical base rates of the
condition in the target population, not the actual base rate of the condition in the
experimental sample.
(3) Always evaluate the clinical utility of a test/measure based on the interpretation of
sensitivity, specificity, PPP, and NPP together. Never interpret sensitivity and specificity
in isolation. Never interpret positive predictive power and negative predicting
power in isolation.

Question 10

What is the main difference in studying test results in NP-research and medical tests?

Answer 10

In NP-research, we mainly focus on finding significant associations and causality (this is often sufficient), while in medical tests this is far from sufficient. To know if results have clinical implications, they also need information on the sensitivity, specifity, CI’s and likelihood rations (more generally speaking, descriptive statistics). P-values and odds ratio’s are, so to say, secondary statistics in medical tests.

Question 11

Which two forms of variability are there in tests results?

Answer 11

1. Intra-observer variability = the lack of reproducibility in results when the same observer/laboratory performs the test on the same specimen at different times.
2. Inter-observer variability = the lack of reproducibility among two or more observers.

Question 12

Studies of reproducability adress (1)….., not (2)…. or (3)…..

Answer 12

(1) precision
(2) accuracy
(3) validity
- –> All observers can agree with each other and still be wrong.

Question 13

What is the basic design to assess reproducibility?

Answer 13

Comparing test results from more than one observer (inter-observer) or that were performed on more than one occasion (intra-observer)

Question 14

How can the inter-observer agreement be measured in categorical variables?

Answer 14

1. The percentage of observations on which the observers agree exactly. Cons: hard to interpret, and it counts partial agreement the same as complete agreement.
2. Kappa measures the extent of agreement beyond what would be expected from observers’ knowledge of the prevalence of abnormality, and can give credit for partial agreement.

Question 15

How can the inter-observer agreement be measured in continuous variables?

Answer 15

This depends on the design of the study; can be done with mean differences between the measurements, SD’s, etc. Another possibility is the Coefficient of Variation (CV) which is the standard deviation of all of the results obtained from a single specimen divided by the mean value.

Question 16

What do studies of the accuracy of tests measure?

Answer 16

They measure (assuming that there is a gold standard) to what extent a test gives the right answer.

Question 17

How can accuracy studies of diagnostic tests be designed ?

Answer 17

Studies of diagnostic test accuracy resemble a case-control or cross-sectional design. These CANNOT be used to estimate predictive values (because the researchers know about the health status before the measurement bc they need to be assigned to the HC or treatment group, causing bias).

Consecutive sampling: more valid/interpretable results ánd predictive values

Tandem testing: compare 2 (presumably imperfect) with eachother to determine which one is more accurate > after that, the gold standard test is selectively applied on positive results + on a random sample of negative results, to make sure they really don’t have the disease.

Cohort designs: necessary for prognostic tests (intervention + follow-ups to see who develops the outcome of interest)

Question 18

What is the outcome variable in diagnostic tests?

Answer 18

The presence or absence of a disease, determined by the gold standard.

Question 19

What is the outcome variable in a prognostic test study?

Answer 19

What happens to patients with the disease (how long they live, complications, additional treatments that they require).

Question 20

What is the ROC curve and how is it used?

Answer 20

Receiver operating characteristic curves; this is a visual representation of the trade-off between sensitivity and specifity of diagnostic tests.

The area under the ROC curve (0.5 = useless, 1 = perfect test) gives a summary of the overall accuracy of the test; useful to compare the accuracy of two or more tests

Question 21

What are likelihood ratios and how are they used?

Answer 21

Also used for continuous/ordinal diagnostic tests to describe their accuracy; better than sensitivity/specifity and ROC (because it uses all information of a test).

The ratio gives the probability of the result given disease (or not). The higher the ratio, the better the test is in ruling in a disease; the lower the ratio (the lower to 0), the better the test result is for ruling out a disease.

Question 22

What is net reclassification improvement?

Answer 22

A more direct approach than ROC curves to quantify what new tests add to existing prediction models.

More specifically, to examine how often a model/clinical prediction rule including the new test changes the classification of patient from one risk category to another, compared to the old model.
–> when new test improves the prediction, more subjects who develop the disorder move up a higher risk category (cases) than move down to a lower risk category (controls)

Question 23

What is the difference between studies to assess accuracy and studies to create clinical prediction rules?

Answer 23

The goal in creating clinical prediction rules is to improve clinical decisions by using mathematical methods to develop a new test, rather than to evaluate one that already exists.

Question 24

How can studies for clinical prediction rules be designed?

Answer 24

Using mathematical models to create prediction rules, such as the multivariate technique (including predictor variables to predict the outcome) such as logistic regression or the Cox (proportional hazards) model, who can quantify the independent contribution of candidate predictor variables for predicting the outcome.

A non-modeling technique is recursive partitioning/Classification and Regression Tree (CART) analysis, which both have very high levels of sensitivity.

Question 25

Why should you validate the clinical prediction rule that you developed?

Answer 25

To avoid overfitting; adress this by dividing the cohort in derivation and validation data sets, and testing the rule derived from the derivation cohort using the data from the validation cohort (this only adressess internal validation). To adress external valididation, you need to test the rule in different populations. So; lack of generalizability.

Question 26

What do diagnostic yield studies assess?

Answer 26

They estimate the proportion of positive tests among patients with a particular indication for the test. Not very useful, but when the study shows that a test is almost always negative, you can conclude that this test is not very useful for the particular indication.

Question 27

What do before/after studies of clinical decision-making assess?

Answer 27

They adress the effect of a test result on clinical decisions. They generally compare what clinicians do before and after obtaining results of a diagnostic test.

Question 28

What do studies do to adress the feasibility, accuracy and validity of tests?

Answer 28

Use descriptive statistics (such as CI’s, means, SD’s, ranges etc)

Question 29

Which design is the best for studying the utility of a diagnostic test? and which variables whould be among the outcomes?

Answer 29

The best design is a clinical trials, where subjects are randomly assigned to receive the test yes/no.

Clinical trials minimize confounding and selection bias and allow measurement of all relevant outcomes (mortality, morbidity, cost and QoL). BUT not practical and ethical issues (withholding potentially valuable tests).

Question 30

Which design can you use to study the utility of a diagnostic test when the clinical trial is not ethical/feasible? To which two problems should you be aware?

Answer 30

Then you can use observational studies (looking at the benefits, harms and costs of the test), with special attention to possible biases (people who volunteer to participate often differ from those who don’t) and confounding (because you don’t have randomisation).

Question 31

What are possible pitfalls in the design/analysis of diagnostic test studies?

Answer 31

1. Inadequate sample size; rare diseases need very largs sample sizes
2. Inappropriate exclusion: it is inapproprate to exclude subjects from the numerator without excluding similar subjects from the denominator
3. Dropping borderline/uninterpretable results: look into why this has happened
4. Verification bias: selective application of a single gold standard, this causes a problem if the test being studied is also used to decide who gets the gold standard.
5. Differential verification bias: different gold standards for those testing positive and negative. This bias can occur any time the gold standard differs among those with positive and negative test results. Can be solved by using the same gold standard for all subjects.

Question 32

How can studies of the value of prognostic tests (focus on the influence of specific variables/risk factors that influence the outcome of the study) be summarized?

Answer 32

Risk ratios, hazard ratios and reclassification improvements

Question 33

What is spectrum bias?

Answer 33

When the spectrum of disease (or non-disease) in the sample differs from that of patients to whom the investigator wants to generalize.

Question 34

Regarding the evaluating of a test, what is often a first good step?

Answer 34

Measuring the reproducability of the test, including the intra- and inter-observer variability.

Question 35

What are three ways to perform internal validation of a clinical prediction model?

Answer 35

1. Split sample: use half of the sample to develop model, test on other half
2. Cross-validation: develop model on randomly drawn half, test on other half
3. Bootstrapping: drawing many samples from original sample of same sample size (drawing with replacement)

Question 36

Where do you use logistic regression/discriminant analyses for in clinical prediction models?

Answer 36

Using these, you can test if the number of prediction variables can be reduces (see when the model reaches significance yes/no, see the added predictive value of independent variables). This can save a lot of time, effort and costs.

Question 37

Regarding logistic regression and discriminant function analysis; which one is preferred and why?

Answer 37

Logistic regression is often preferred, because this has fewer assumptions as discriminant function analysis; regarding the rest, they actually do the exact same thing.