Statistics Flashcards
Cement your grasp of certain fundamental concepts in binary classification and inferential statistics: true and false positives and negatives, positive and negative predictive values, ROC curves and AUC, Bayes' Theorem, and p-values.
In an experiment with a high p-value, your data are highly ____ given a true null hypothesis.
likely
In an experiment with a low p-value, your data are highly ____ given a true null hypothesis.
unlikely
The p-value is the probability of…
obtaining an effect AT LEAST as extreme as the one observed assuming that the null hypothesis is true
A study found a difference between two means with a p-value of 0.02. Interpret this p-value in terms of many repetitions of identical studies.
If you repeated the study many times, you would find differences at least as large as observed in this study 2% of the time.
The p-value answers what question?
How likely are your data given that the null hypothesis is true?
The probability of falsely rejecting a true null hypothesis is called…
Type I error = false alarm = false positive
What two factors determine the probabilities of Type 1 and Type 2 errors?
The desired level of significance and the power of the test
The probability of falsely accepting a false null hypothesis is called…
Type II error = missed detection = false negative
Does type 1 error reject or accept the null hypothesis?
Type 1 error (odd number) rejects the null hypothesis, an “even” number
Does type 2 error reject or accept the null hypothesis?
Type 2 error (even number) accepts the null hypothesis, another even number
If you commit a type I error, what do you do to the null hypothesis?
Type I error = reject the null hypothesis even though it’s actually true = false positive
If you commit a type II error, what do you do to the null hypothesis?
Type II error = accept the null hypothesis even though it’s actually false = false negative
A false positive is also known as what kind of error?
Type I: false Positive has one vertical line
A false negative is also known as what kind of error?
Type II: false Negative has two vertical lines
What’s the formula for statistical power in terms of α and/or β?
power = 1 - β
If an experiment’s probability of type II error increases, then the statistical power ____
decreases; power = the ability to correctly reject a false null hypothesis = 1 - β
The likelihood that a study will detect an effect when there really is one to be detected is…
statistical power
A study reports no effect when in fact there was one. What kind of error is this?
Type II error = β = false negative
A study reports an effect when in fact there was no effect. What kind of error is this?
Type I error = α = false positive
What four factors affect statistical power?
- effect size
- sample size
- desired α (type I error)
- the chosen or implied β, or equivalently, the statistical power 1 – β
Given any three of these, you can find the fourth.
What are the two families of effect size indexes?
- differences between groups (risk ratio, odds ratio, Cohen’s d, Glass’s delta, etc.)
- measures of association (corr coeff r, r^2, Spearman’s rho, Cohen’s f, etc.)
T or F: the p-value is the probability of getting a false positive.
False! p = probability of seeing at least that big an effect, assuming null is true. It is actually impossible to calculate the probability that the null hypothesis is true solely from sample statistics.
In general, which is greater: the probability that the null hypothesis is true, or the p-value?
The probability that null is true tends to be greater than the p-value by a large margin.
T or F: a confidence interval is a range of values that is likely to contain an unknown population parameter.
True
If you draw a random sample many times, a certain percentage of the confidence intervals will contain the population mean. What is the name for this percentage?
The confidence level
T or F: the confidence LEVEL is the probability that a specific confidence interval contains the population parameter.
False! For any given study, the confidence interval either contains or does not contain the population parameter of interest.
Express confidence level in terms of α and/or β.
Confidence level = 1 – α
If the confidence interval does not contain your null hypothesis value, what can you say about statistical significance?
If CI does not contain the H_0 value, the result is statistically significant.
If p < α, what do you know about the confidence interval?
If p < α, the confidence interval will not contain the null hypothesis value.
If 95% confidence intervals for two independent sample means overlap, could there be a statistically significant difference between them?
Yes! Non-overlapping CIs always imply a significant difference. But 95% CIs can sometimes overlap even when p < 0.05.
What is the definition of positive predictive value?
PPV is the probability that a significant result represents a true effect. In terms of disease screening, PPV = P(disease | positive).
In the context of disease screening, what is PPV expressed as a conditional probability?
PPV = P( disease | positive).
In the context of disease screening, what is NPV expressed as a conditional probability?
NPV = P(healthy | negative).
In the context of disease screening, what is the sensitivity expressed as a conditional probability?
Sensitivity = P(positive | disease).
Recall: sensitivity = true positive rate
In the context of disease screening, what is specificity expressed as a conditional probability?
Specificity = P(negative | healthy).
Recall: specificity = true negative rate
T or F: specificity is synonymous with negative predictive value (NPV).
False.
Specificity = P(negative | healthy), whereas NPV = P(healthy | negative).
T or F: sensitivity is the same thing as positive predictive value (PPV).
False.
Sensitivity = P(positive | disease), whereas PPV = P(disease | positive).
What does a ROC curve have on its x- and y-axes?
x-axis: FPR = 1 – specificity = α
y-axis: TPR = sensitivity
(Sensitivity = true positive rate, and Specificity = true negative rate)
What is another term for the sensitivity of a binary classifier?
True positive rate
What is another term for the true positive rate of a binary classifier?
Sensitivity
What is another term for the specificity of a binary classifier?
True negative rate
What is another term for the true negative rate of a binary classifier?
Specificity
Given the true positive rate, how do you calculate the false negative rate?
TPR = 1 – FNR
Think: true positives (TPR) + positives falsely labeled as negatives (FNR) = 1.
How do you calculate the true positive rate if you know the false negative rate?
FNR = 1 – TPR
Think: true positives (TPR) + positives falsely labeled as negatives (FNR) = 1.
What is the false positive rate in terms of specificity?
FPR = 1 – specificity.
FPR = 1 – TNR, since specificity = true negative rate.
