Chapters 16 and 17 Flashcards
What does Statical Hypothesis Testing do?
It automates decision-making and favors objectivity by setting a threshold a priori.
a (alpha):
used to designate the significance threshold
What do you conclude and therefore do if a P value is less than the alpha?
Conclude that the difference is statistically significant and reject the null hypothesis
Type I Error
Incorrectly reject null
Type II Error
Incorrectly fail to reject null
Is there an option to accept null?
No, just to reject or not
Which type of error is preferred?
Usually, a type II error is preferred over type I error. e.g. prefer to release a criminal over imprisoning innocents
Choose alpha how?
a priori
Which alpha is common?
0.05 is common
Increasing alpha makes what more likely?
Increasing alpha makes type I more likely and type II less likely.
Decreasing alpha makes what more likely?
Decreasing alpha makes type II more likely and type I less likely
What kind of alpha will a very conservative scientist choose?
A low alpha, however, scientists often just report the exact P value and put less emphasis on choosing alpha.
You will never know…
from a single analysis if you have made a type I or type II error.
Even worse error than type I or type II.
Rejecting the null hypothesis correctly but wrong about effect direction.
The only way to reduce the chances of all three errors is…
to increase sample size.
Just because a result if statistically significant does not ensure…
that it is scientifically important.
The ______ may not be worthy of attention.
size of the effect.
Common mistakes
- Believing that statistical hypothesis testing is an essential part of all statistical analyses
- Could just look at P value and CI without alpha
Common mistakes 2
- Believing that if a result is statistically significant, the effect must be large.
- Focus on the size of the effect and its CI
If the sample size is large enough, even tiny, inconsequential differences will be…
statistically significant.
Common Mistakes 3
If the results are non-significant with the analysis that you chose, you can’t just do a slightly different analysis (e.g. nonparametric) with the same data and report only the most unusual results
- You shouldn’t just add more data if you got close but not quite significant results (must redo all of it)
- Probably a common violation
Don’t do multiple tests and just…
report the significant ones (No data diving)
- Possibly the cause of many conflicting studies and reporting significant but incompatible results
- If many separate labs study the same problem then this can happen inadvertently
If the 95% CI includes the value that defines the null hypothesis, you can conclude the P value is…
> 0.05
If the 95% CI excludes the null hypothesis value, then the P value is…
The size of the CI and the range of the nonsignificant values are the same size if confidence level=
1-alpha
