Lecture 14: Null Hypothesis Testing Flashcards
The alpha level
How low the p-value has to be for us to reject the null hypothesis (alpha)
Conventionally what alpha level is used?
0.05
The alpha level is usually 1 minus the _______ ____
Confidence level
Null hypothesis
There is no effect
Alternative hypothesis
There is an effect
Typically the research expects/hopes that the null hypotheis is TRUE/FALSE
FALSE! So that the alternative hypothesis is true (we’d want to hope that the drug would work)
Before we collect our data we want to set the _____ ______
Alpha level
The alpha level defines how much ________ we need for our null hypothesis test
CONFIDENCE!
After we get our data, the null hypothesis test is a 3 step process (3)
- Compute the test statistics and degrees of freedom
- Compute a p-value from the test statistic and df
- Compare the p-value to a
If p < a then we do we accept or reject the null hypothesis?
REJECT! There IS sufficient evidence of an effect!
if p > a do we reject/accept the null hypothesis?
ACCEPT! There ISN’T ENOUGH sufficient evidence of an effect
For mean comparisons, the null hypothesis test we typically do is called a _________
T-test
There are 2 kinds of 2 tests!
- Paired (dependent samples) t-tests
- unpaired (independent samples) t tests
If we use a pooled s we call it a ______ t-test
Student
If we use an unspooled s we call it a ____ t-test
Welch’s
How does the t-test work? (3)
- Compute test statistics and df
- Compute a p-value from the test statistic and df
- Compare the p-value to the alpha level (a)
What’s a short term for the test statistic?
T-statistic
What’s a t-statistic? How do we compute this?
The sample mean difference OR AKA estimated effect size /estimated standard error
What is the df based on?
Sample size
For a paired comparison the df =?
Df = n-1 (n being the number of pairs)
For unpaired comparisons the df =?
Df kinda = n-2
For step two of the t-test it consists of a complex computation we’d never do by hand BUT basically the larger the t (in absolute value) then the smaller the _____
P
If the p < a we say that the p-value and the sample mean difference that produced it are ______ __________
Statistically significant
Why is the term “significant” critiqued?
It implies importance by even a small difference can be likely to be statistically significant if the sample size is large
Saying that p < a is also equivalent to saying that the confidence interval (at the 1-a confidence level) doesn’t include ____
Zero
Since 95% confidence intervals don’t contain the actual parameter in 5% of the studies we do how often will a p-value be less than 0.05 when the actual population mean difference is 0?
5% of studies when the population mean difference is zero
For a true null hypothesis the p-value is _______ distributed between ___ and ____
Uniformly distrusted between 0 and 1
All values are equally likely
For a normal distribution, the probably of p < 0.05 is ____% and so is the probability of p > 0.95 and so is the probability of p being in between 0.65 and 0.7
5%
When the null hypothesis is false the chance of getting a p-value less than 0.05 are ______ which means that the skew is _____
Increased! Positive!
What two things, if they were bigger, would skew the p-value distribution to the right? (2)
- Standardized effect
- Sample size
For a true null hypothesis, the p-value is uniformly distributed between ___ and ___
0 and 1 = no value is more likely than another regardless of sample size
For a false null hypothesis, the ___ p-values are more likely
low p-values = bigger the effect, bigger the sample size, lower p-value is likely to be
Rejecting a true null hypothesis is which error?
Type 1 error (or aka type 1 error rate)
Why not set the alpha level to be very low instead of at 0.05?
Then we would rarely reject the null hypothesis even when it’s false…
Not rejecting the null hypothesis when it’s false is which error?
Type 2 error
So a type 1 error is finding an effect that ____ there in the population
isn’t realy there
A type 2 error is basically ______ to find an effect that is really _____ in the population
failing to find an effect that is really there in the population
The lower we set the alpha level the ____ the type 2 error rate if the null hypothesis is false
Higher
How can we use an alpha level that’s sufficiently low without risking an excessively high type 2 error rate?
LARGE SAMPLE!!!
If we use a large sample we can set a ___ alpha level and still have a good chance of getting statistical significance is the null hypothesis is false
Low!
So what protects us from type 1 errors if the null hypothesis is true?
LOW ALPHA LEVELS!
So what protects us from type 2 errors if the null hypothesis is false?
LARGE SAMPLE SIZE!
Statistical power (and the formula)
The probability of getting statistical significance when the null hypothesis is false
1 - Type 2 error rate
What makes statistical power high?
- Large standardized effect size
- Large sample size
