Hypothesis Testing Flashcards
What are the hypotheses in a statistical test?
The NULL hypothesis H0, is the claim claim that is initially assumed to be true (the “prior belief” claim).
The ALTERNATIVE hyothesis Ha is the assertion that us contradictory to H0.
The null hypothesis will be REJECTED in favor of the alternative hypothesis ONLY IF SAMPLE EVIDENCE suggests that H0 is FALSE.
If the sample does NOT STRONGLY contradict H0, we will CONTINUE to BELIEVE in the TRUTH of the NULL hypothesis.
The two possible conclusions from a hypothesis testing analysis are then [REJECT H0] or [FAIL to REJECT H0].
Define the error types in hypothesis testing.
A type I error consists of REJECTING H0 when it is TRUE (false negative).
A type II error involves NOT REJECTING H0 when it is FALSE (false positive).
Changing the decision threshold leads to different probabilities of making type I or type II errors. These probabilities are traditionally denoted by alpha and Beta, respectively.
Because H0 specifies a unique value of the parameter, there is a single value of alpha.
However, there is a different value of Beta for each value of the parameter consistent with Ha.
A certain car is said to sustain no damage in 25% of all 10 mph crash tests.
A modified bumper has been proposed to increase this percentage.
Let p denote the proportion of all 10 mph crashes with this new bumper that result in no physical damage.
The hypothesis to be tested are:
H0: p = .25
Ha: p > .25
The test will be based on an experiment with n=20 independent crashes with prototypes of the new design.
H0 should be rejected if a substantial number of the crashes shows no damage.
Consider the following test procedure:
test stat: X = {# of crashes with no damage}
reject region: R8 = {8,9,…,20}, i.e. reject H0 if x>=8.
When H0 is true, X has a BINOMIAL PROBA DIST with n=20 and p=.25. Then
alpha = P(type I error) = P(H0 is rejected when it is true)
= P(X>=8 when X~Bin(20,.25) = 1-B(7;20,.25)
= 1 - .898 = .102
i.e., when H0 is actually true, roughly 10% of all experiments consisting of 20 crashes would result in H0 being incorrectly rejected (a type I error).
In contrast to alpha, there is not a single Beta. Instead there is a different Beta for ea different p=.3 (in which case X~Bin(20,.03), another for p=.5, and so on.
e.g. Beta(p=.3) = P(type II error when p=.3) = P(H0 is not rejected when it is false because p=.3) = P(X<=7 whe X~Bin(20,.03) = B(7;20,.3) = .772
When p is actually .3 RATHER than H0: p=.25, a “small” departure from H0, roughly 77% of all experiments of this type would result in H0 being INCORRECTLY REJECTED.
What is power of a statistical test?
The power of a hypothesis test is the PROBABILITY that the test REJECTS the NULL hypothesis when the ALTERNATIVE hypothesis Ha is TRUE.
i.e. we do the right thing, we do the opposite of type II error of FAILING to reject H0 when it is FALSE.
The value of power ranges from [0,1].
recall error types:
type I: H0 is TRUE, but IS rejected (false pos)
type II: H0 is FALSE, but is NOT rejected (false neg)
As power INCREASES, the probability of making a TYPE II error (INCORRECTLY FAILING to reject the NULL hypothesis) DECREASES.
recall error probabilities:
alpha: type I error probability (FP proba)
Beta: type II error probability (FN proba)
For a type II error probability Beta, the corresponding statistical power is 1-Beta.
Power can be thought of as the probability of ACCEPTING the NULL hypothesis when it is in fact TRUE, i.e. the ability of a test to actually detect an effect if that specific effect ACTUALLY exists.
Then,
Power = P(reject H0 | Ha is TRUE)
e.g.
experiment 1 has a power of 0.70
experiment 2 has a power of 0.95
Then there is a stronger probability that experiment 1 had a type II error than experiment 2
How does a normal Z table relate to p values?
When you test a hypothesis about a population, you can use your test statistic to decide whether to reject the NULL hypothesis H0.
You make this decision by coming up with a a number, called a P-VALUE.
A p value is a PROBA ASSOCIATED with your CRITICAL alpha value. The critical value depends on the PROBABILITY you are ALLOWING for a TYPE I error, i.e. rejecting H0 when it is true.
p value measures the probability of getting results at least as strong as yours if the claim H0 was true.
Imagine a standard normal pdf (and cdf under the pdf). Alpha critical values are the values at the tails of the standard normal curve. We REJECT H0 if the Z value is BEYOND the alpha critical level on the x-axis, and we FAIL to reject H0 if the Z value falls short of the extrme alpha levels (Z is in t he middle non-tail region of the std normal curve).
