Stats - statistical test comparisons and ROC curves Flashcards
What does ROC curve stand for and what is it useful for?
ROC (receiver operating characteristic) curves are used to tell how good a test is at distinguishing between two things (e.g. which patients have a disease and which don’t), and to help decide on the threshold that should be used.
The whole point of an ROC curve is to help you decide where to draw the line between normal and not normal.
What are 4 things an ROC curve can do
?
What are their main purpose in psychiatry?
1) evaluate the discriminatory ability of a continuous marker to correctly assign into a two-group classification
2) find an optimal cut-off point to least mis-classify the two-group subjects
3) compare the efficacy of two (or more) diagnostic tests or markers
4) study the inter-observer variability when two or more observers measure the same continuous variable
(In psychiatry, ROC curves are usually used to assess the performance and utility of diagnostic / screening tools).
How is an ROC curve designed?
What are the axis and what does it measure?
A ROC curve is created by plotting the true positive rate against the false positive rate at various threshold settings (see below). This is done plotting sensitivity (true positive rate) against 1-specificity (false positive rate).
The dot closest to the top left hand corner is the one with the best trade off between sensitivity and specificity.
What does the AUC measure?
What would it be if the test were 100% sensitive and specific?
Diagnostic accuracy
If a test had a sensitivity of 1 (100% sensitive) and a specificity of 1 (100% specific) then the area under the curve would be 1. Therefore, it follows that the higher the AUC is the better the overall performance of the test (i,e. the higher the accuracy).
In conventional grading of AUCs, what values correspond to the following labels?
Excellent
Good
Fair
Poor
Fail
0.9-1 Excellent
0.8-0.9 Good
0.7-0.8 Fair
0.6-0.7 Poor
0.5-0.6 Fail
What are the 5 steps to follow in order to establish the best statistical test to use?
Step 1 → Define the research question (correlation or difference)
Step 2 → Establish how many groups there are and if dependent or independent
Step 3 → Identify the variables and data types
Step 4 → Establish if parametric or non-parametric tests should be used
Step 5 → Select the correct statistical test (as per chart below)
What is meant by each of the steps in choosing an appropriate statistical test?
Step 1 → Define the research question (correlation or difference):
Correlation/Relationship: Are you exploring whether variables are associated or related?
Example: Is there a relationship between smoking status and anxiety levels?
Difference Between Groups: Are you comparing two or more groups to identify differences in outcomes?
Example: Does a new therapy reduce depression more effectively than standard therapy?
What is meant by each of the steps in choosing an appropriate statistical test?
Step 2 → Establish how many groups there are and if dependent or independent:
When we have independent samples, we assume that the scores of one sample do not affect the other. Each group contains different subjects and there is no meaningful way to pair them.
Dependent samples are related to each other. Dependent samples can occur in two scenarios. In one, a group may be measured twice such as in a pretest-posttest situation (scores on a test before and after the lesson).
The other scenario is one in which an observation in one sample is matched with an observation in the second sample.
For example, suppose quality inspectors want to compare two laboratories to determine whether their blood tests give similar results. They send blood samples drawn from the same 10 children to both labs for analysis.
Because both labs tested blood specimens from the same 10 children, the test results are not independent. To compare the average blood test results from the two labs, the inspectors would need to do a paired t-test, which is based on the assumption that samples are dependent.
One Group:
Are you testing a single group against a known value or hypothesis?
Example: Is the mean depression score in a single group significantly different from the national average?
Two Groups:
Independent: Groups consist of separate individuals (e.g., treatment vs. placebo groups).
Example: Do smokers and non-smokers differ in their anxiety levels?
Dependent/Related: Groups consist of the same individuals tested under different conditions or time points.
Example: Do anxiety scores change before and after therapy in the same group of participants?
Three or More Groups:
Independent: Separate groups (e.g., non-smokers, light smokers, heavy smokers).
Dependent/Related: Repeated measures or matched groups (e.g., blood pressure measured at three time points).
What is meant by each of the steps in choosing an appropriate statistical test?
Step 3 → Identify the variables and data types:
Dependent Variable: The outcome you are measuring (e.g., anxiety score).
Independent Variable: The factor you are comparing or assessing (e.g., treatment type, smoking status).
A variable is any characteristics, number, or quantity that can be measured or counted. A variable may also be called a data item. Age, sex, business income and expenses, country of birth, capital expenditure, class grades, eye colour and vehicle type are examples of variables. It is called a variable because the value may vary between data units in a population.
The three main variables in a typical study are the, independent, dependent and controlled variables.
The independent variable is something that the experimenter purposely changes over the course of the investigation.
The dependent variable is the one that is observed and changes in response to the independent variable.
During the experiment all other variables should be controlled. The variables that are not changed are called controlled variables.
Dependent variables are affected by independent variables but not by controlled variables as these do not vary throughout the study.
Data type:
Nominal
Ordinal
Ratio
Continuous
What is meant by each of the steps in choosing an appropriate statistical test?
Step 4 → Establish if parametric or non-parametric tests should be used:
Parametric tests are valid if the following 4 assumptions are met:
- Normal Distribution:
The data should follow a bell-shaped (normal) distribution.
Applies to continuous (interval/ratio) variables. - Homogeneity of Variance:
Variability (spread) within each group should be approximately equal.
Particularly relevant for tests comparing multiple groups (e.g., t-tests, ANOVA). - Independent Observations:
Data points should be independent of each other (no pairing or repeated measures, unless explicitly accounted for). - Scale of Measurement:
The dependent variable must be interval or ratio.
What is meant by each of the steps in choosing an appropriate statistical test?
Step 5 → Select the correct statistical test
Use the information from the previous steps to choose the best test (see diagram)
When can you use the Chi-squared test?
Is it parametric or non-parametric?
What assumption is made?
Used to assess differences in categorical variables
Non-parametric test
Applies an assumption that the sample is large (Fisher’s exact does not make this assumption and so should be used for small samples although it can be used for both large and small samples)
Compares the observed frequencies against those that would have been expected if there was no difference and then produces a value (the chi squared) which can then be used to assess if the difference is significant (p<0.05).
When would you use a T-test?
What are the 3 types of t-tests?
Is it parametric or non-parametric?
A t-test is used to assess whether the means of two groups have a statistically significant difference.
There are 3 types of t-test:
- one sample t-test (this is used to see if there is a difference between a sample mean and a hypothesised population mean (or claimed mean)
- 2 samples paired T-test
- 2 samples independent T-test
T-tests are PARAMETRIC
What is ANOVA?
How is it similar or different from T-tests?
Is it parametric or non-parametric?
ANOVA is a statistical test to demonstrate statistically significant differences between the means of several groups. It is similar to a student’s t-test apart from that ANOVA allows the comparison of more than just two means.
It is a PARAMETRIC test
What is the non-parametric equivalent of the following tests:
Independent T-test
Paired T-test
Mann Whitney-U (NP for 2 independent groups) (men are independent)
Wilcoxon (NP for 2 dependent groups) - (I WILl DEPEND on you)
