Chapter 21: More About Tests Flashcards
Define ‘Statistically significant’.
When the P-value fells below the alpha level, we say that the test is “statistically significant” at that alpha level.
Implies strong evidence that null is false, not that the difference is important.
Define ‘Alpha level (α)’.
The threshold P-value that determines when we reject a null hypothesis. If we observe a statistic whose P-value based in the null hypothesis is less than α, we reject that null hypothesis.
Define ‘Significance level’.
The alpha level is also called the significance level of a test, most often in a phrase such as a conclusion that a particular test is “significant at the 5 % significance level”.
Define ‘Critical value’.
The value in the sampling distribution model of the statistic whose P-value is equal to the alpha level. The critical value is often denoted with an asterisk, as z*, for example.
Define ‘Type I error’.
The error of rejecting a null hypothesis when in fact it is true (called a “false positive” in diagnostic tests). The probability of a Type I error is α.
Define ‘Type II error’.
The error of failing to reject a null hypothesis when in fact it is false (called a “false negative” in diagnostic tests). The probability of this is β (not one number, but a function of the alternatives to the null).
Define ‘β’.
The probability of a Type II error. β depends on the specified alternative value for the parameter (or effect size).
Define ‘Power’. (Particularly concerned with power when we fail to reject a null hypothesis)
The probability that a hypothesis test will correctly reject a false null hypothesis is the power of the test (for some specified effect size).
To find power, we must specify a particular alternative parameter value as the “true” value. For any specific value in the alternative, the power is 1 - β.
Define ‘Effect size’.
The difference between the null hypothesis value and true value of a model parameter.
A larger effect size should be easier to test for.
The null hypothesis specifies..?
The alternative hypothesis specifies…?
A parameter and a (null) value for that parameter.
A range of plausible values should we fail to reject the null.
A P-value is…? What does a small/large P-value indicate?
The estimated probability of observing a statistic value at least as far from the (null) hypothesized value as the one we have actually observed.
A small P-value indicates that the statistic we observed would be unlikely were the null hypothesis true. Leads us to doubt null.
A large P-value tells us that we have insufficient evidence to doubt null (however, does not prove null true).
A test may be statistically significant, but practically meaningless if the estimated ___ is of trivial importance.
Effect (refer to section with overlapping diagrams).
Increasing the sample size will generally improve the ___ of any test.
Power.
