Statistical theory 3 Flashcards
What is null hypothesis significance testing called for frequentists?
orthodox hypothesis testing
or, null hypothesis significance testing (NHST)
What was Fisher’s thoughts on hypothesis testing?
Hypothesis testing is about trying to falsify a single hypothesis
What was Neyman’s theory on hypothesis testing?
Hypothesis testing is about choosing between two rival hypotheses
What does the null hypothesis predict?
There will be no effect
In null hypothesis testing, what is H1?
The alternative hypothesis
This one is looking for some effect (usually the one they want to find)
Do we always test the hypothesis or null hypothesis?
Null hypothesis
What is a statistical hypothesis?
Claims about population parameters
“A critical part of designing a study is being able to specify how your 1)_____ hypothesis (what you’re interested in) maps onto your 2)______ hypothesis (what you can actually test)”
1) Research
2) Statistical
How do we evaluate whether the null or alternative hypothesis is accurate?
By determining how likely our data would be if the null is true.
If they are unlikely ‘enough’, we reject the null
In hypothesis testing, what is a Type 1 error?
False positive
In hypothesis testing, what is a Type 2 error?
False negative
What type of error rate does null hypothesis testing control for?
Type 1 error rate
Why is null hypothesis significance testing designed the way it is?
To control for belief bias (avoid type 1 errors)
Forces a minimum evidentiary standard on you
What is the significance level?
If we would expect to see our data n% of the time (or less) if the null were true, we reject the null
The choice for n is arbitrary and is known as the significance level
What do you need to build your own statistical test?
Important for all types of statistical tests
A diagnostic test statistic T
Sampling distribution of T if the null is true
The observed T in your data
A rule that maps every value of T onto a decision (accept or reject H0)
What is a diagnostic test statistic (T )?
A single number that you can calculate only from your observations
As long as it’s just one number, it’s a test statistic
What are these all examples of?
- The mean of a set of observations*
- The standard deviation of a set of observations*
- The third-largest of a set of observations*
- A number of observations*
Diagnostic test statistics
Is the two largest numbers in the sample an example of a test statistic?
No
(multiple numbers)
How do you calculate sampling distribution if the null is true?
1st. Assume H0 is true
Figure out what the values of your test statistic you should expect
A test statistic is diagnostic if the null hypothesis and alternative predict _____ values
different
- e.g. Baby v cat cuteness: number of baby choices out of 100*
- H0: Should be around 50*
- H1: Should* not be around 50
In a binomial test statistic, when should we reject the null?
When 95% of the data accepts the H0 and the results lie outside of that.
Where is the ‘rejection region’ for null hypothesis testing?
Outside of the 95% probability range.
95% probability means that if the null hypothesis is true, there is a 5% chance of _____ rejecting it.
falsely
What is a ‘two-tailed’ test?
A directionless test
When the rejection region for the null hypothesis lies on both sides of the distribution
What is a ‘one-tailed’ test?
A directional test
Rejection region only covers one tail (still 5%)
Why do we use 5% for a desired Type 1 error rate?
α = .05 is the default significance level that we can use in science, can also be α = .01 or α = .001
What is a p-value according to Neyman (null hypothesis guy)?
p describes the Type 1 error rate you must be willing to tolerate if you want to reject H0
p is the smallest significance level your data would let you adopt while still being able to reject the null
What is a p-value according to Fisher?
p is the probability - if H0 is true - of observing a test statistic at least as extreme as the one that was actually found
A measure of how implausible the data are, according to H0
When running a null-hypothesis hypothesis test, what must we do?
Adopt a ‘standard’ significance level of α = .05
If p < α reject the null hypothesis
Otherwise, accept (or retain) the null hypothesis
Is the phrase “p is the probability that the null hypothesis is true” accurate?
No, never say this
- P-value is a claim about how likely you were to see your data if the null hypothesis were true. This is not the same thing as a claim about whether the null hypothesis is true*
- A claim about H0 is true depends on what other hypotheses you’re considering, for that you need to be able to evaluate them.*
How do you report a p-value?
Which of the following groups of phrases are correct?
1) “The null hypothesis is true”
“The alternative hypothesis is false”
We have “proved” that…
Or
2) “We retain the null hypothesis”
“We failed to reject the null hypothesis”
“The test was not significant”
(“Accept” the null)
Or
3) “We reject the null hypothesis”
“The test was significant”
(“Accept” the alternative)
2 and 3
1 are all very definitive statements, they imply we know the truth but we do not know the truth
