Stats - statistical test comparisons and ROC curves

Question 1

What is the first question we need to ask about our data to determine the appropriate statistical test?

Answer 1

(1) type of predictor variable (2 choices)

• nominal or ordinal (Categorical) - includes binary outcomes e.g survival/ not
• ratio or interval (quantitative)

Question 2

Once we have decided if the dependent variable is Qualitative OR Categorical, what is the next question we have to ask for each?

What type of test do we choose for each option (x4)

Answer 2

Same question again - categorical or quantitative

If:
Categorical the

Question 3

What are the three common types of parametric test?

Answer 3

Regression
Comparison
Correlation

Question 4

What does ROC curve stand for and what is it useful for?

Answer 4

ROC (receiver operating characteristic) curves are used to tell how good a test is at distinguishing between two things (e.g. which patients have a disease and which don’t), and to help decide on the threshold that should be used.

The whole point of an ROC curve is to help you decide where to draw the line between normal and not normal.

Question 5

What are 4 things an ROC curve can do
?

What are their main purpose in psychiatry?

Answer 5

1) evaluate the discriminatory ability of a continuous marker to correctly assign into a two-group classification
2) find an optimal cut-off point to least mis-classify the two-group subjects
3) compare the efficacy of two (or more) diagnostic tests or markers
4) study the inter-observer variability when two or more observers measure the same continuous variable

(In psychiatry, ROC curves are usually used to assess the performance and utility of diagnostic / screening tools).

Question 6

How is an ROC curve designed?
What are the axis and what does it measure?

Answer 6

A ROC curve is created by plotting the true positive rate against the false positive rate at various threshold settings (see below). This is done plotting sensitivity (true positive rate) against 1-specificity (false positive rate).

The dot closest to the top left hand corner is the one with the best trade off between sensitivity and specificity.

Question 7

What does the AUC measure?
What would it be if the test were 100% sensitive and specific?

Answer 7

Diagnostic accuracy

If a test had a sensitivity of 1 (100% sensitive) and a specificity of 1 (100% specific) then the area under the curve would be 1. Therefore, it follows that the higher the AUC is the better the overall performance of the test (i,e. the higher the accuracy).

Question 8

In conventional grading of AUCs, what values correspond to the following labels?

Excellent
Good
Fair
Poor
Fail

Answer 8

0.9-1 Excellent
0.8-0.9 Good
0.7-0.8 Fair
0.6-0.7 Poor
0.5-0.6 Fail