Session 8 - P-values and Significance Testing Flashcards
What is the p-value?
Formally and Informally
-Formally
The probability of seeing something at least as extreme as the data we have just observed, if the null hypothesis, H0, were true.
-Informally
The probability of seeing something at least as interesting by random chance if there were actually no effect.
What are the two types of errors that could be made?
False Positive (“Type 1 Error”):
The incorrect rejection of a true null hypothesis
The chance of a type 1 error happening is denoted by α
Conventionally we use α=0.05
False Negative (“Type 2 Error”)
The failure to reject a false null hypothesis
The chance of a type 2 error happening is denoted by β
The power of a study is 1-β (“if there is a true effect, will we see it?”)
Conventionally we use β=0.2 or β=0.1
The analysis of a clinical trial results in a p-value of 0.0012. How would you report it?
“Reject the null hypothesis”
“There is a statistically significant difference between outcomes in the two arms of the trial”
“If there were truly no difference in outcome between drug and placebo, 0.12% of the time we would see a difference in our sample at least as extreme as the one we have seen here.”
The analysis of a clinical trial results in a p-value of 0.12. How would you report it?
Do not reject the null hypothesis
“There is not a statistically significant difference between outcomes in the two arms of the trial”
“If there were truly no difference in outcome between drug and placebo, 12% of the time we would see a difference in our sample at least as extreme as the one we have seen here.”
What is the Bonferroni Correction?
It is using a lower significance threshold for each test, in order to control the overall chance of making a Type 1 Error
The Bonferroni correction: if you are doing n tests, use the threshold:
a=0.05/n
What affects power?
The magnitude of the effect size: the larger the true effect, the more likely we are to detect an effect, and the higher the power.
The variance in the outcome measure: the greater the variance, the harder it will be to detect an effect, and the lower the power.
The significance threshold chosen: there is a payoff between false positive rate and false negative rate. If we decrease the significance threshold (to decrease the false positive rate) we will increase the false negative rate and decrease the power.
The sample size: effects are harder to detect in smaller samples. A larger sample size will increase the power to detect the effect. In practice, this is the variable we can change a study. A study chooses its sample size to obtain a desired significance threshold and power.
PONDER WONDER
Was there a difference in survival time between SEARCH study participants whose breast cancer was poorly differentiated compared with those whose cancer was not poorly differentiated (i.e. either well or moderately differentiated)?
