L6 Flashcards
What does this show us?
Statistical power in signal detection terms
What is the multiplicity problem?
Running multiple tests of the same null hypothesis can lead to more errors
Dealing with multiple comparisons or tests with a null hypothesis
What is the hidden data problem?
We dont have all the data available to measure the size or effect in reality.
What is the reproducibility problem?
The prevalence of false positives in the literature is higher than we might expect
(reproducibility crisis)
What is a two-tailed test?
The 5% cutoff (alpha) is split evenly across the two ends (“tails”) of the null distribution. Each tail containing 2.5% of the area under the curve
When should we use a two-tailed test?
When we don’t have a strong theoretical explanation about the direction of an effect and are unsure which way it could go.
What is a one-tailed test?
The 5% rejection region is concentrated at one end or “tail” of the null distribution - one tail contains 5% of the total area under the curve.
When would we use a one-tailed test?
We a strong reason to believe that the relationship is only one way.
If the results are the other way, the outcome is meaningless.
What happens if researchers change from a one-tailed to a two-tailed test after looking at the data?
What is this called?
You increase the alpha level from 5% to 7.5%.
We have conducted 2 tests (first test 5% alpha, 1 tail), if we test again again for a 2 tailed we add another 2.5% on the back end and the alpha is now 7.5%
This is called the problem of multiplicity
What is the hocus pocus trick?
When you conduct many experiments where the majority fail, but then due to the 5% alpha a few succeed and you only reveal those ones.
What is the family wise error rate?
Tests that are used to take the error rate of all tests into account to avoid the problem of multilicity.
What is the name of the most popular and conservative family wise error rate calculation?
Bonferroni method
When would we conduct the Bonferroni method during an experiment
Post-hoc
We do the adjustment after the tests
What are the 4 steps for dealing with multiplicity?
- Clearly define the null hypothesis.
- Focus analyses on the most important comparisons.
- Treat post-hoc comparisons as an exploratory analysis.
* Dealing with Multiplicity* - Run follow-up confirmatory experiments with planned contrasts.
What is selection bias?
when the outcome of an experiment or research study influences the decision to publish it.
only publishing positive results and not negative ones
What is the file drawer problem?
That journals are filled with the 5% of studies that show Type 1 errors, while the drawers back at home are filled with the 95% of studies that show nonsignificant results
Selection Bias
What is Inflation Bias (“p-Hacking”)
when researchers try out several statistical analyses and selectively report those that produce significant results.
What are 5 ways you can p-hack your results?
1) stopping data collection after finding a significant p-value
2) recording many DV’s and only reporting DV’s that yield a significant p-value
3) including or dropping outliers to yield a significant p-value
4) excluding, combining, or splitting conditions to yield a significant p-value
5) including or excluding covariates until yielding a significant p-value
What is the texas sharpshooter fallacy
If you are making your hypothesis after finding your results
(an example of p-hacking or HARKING)
Explain the hidden data problem
We ideally would like to analyse all the data (left image) but in reality we only have access to a limited scope of data (right image) and have to infer our results
What are the two classifications of missing data?
Missing at random (values in data set are missing at random)
Missing not at random (value of the variable thats missing is related to the reason its missing)
What is the most problematic type of missing data?
Missing not at random
What are the two ways to deal with missing data?
Imputation (missing data is “filled in”, imputed or replaced with substituted values)
Complete case deletion (all cases with a missing value are deleted)
What is the ceiling/floor effect?
When an independent variable no longer has an effect on a dependent variable, or the level above/below which variance in an independent variable is no longer measurable.
