L6 Flashcards
What does this show us?
Statistical power in signal detection terms
What is the multiplicity problem?
Running multiple tests of the same null hypothesis can lead to more errors
Dealing with multiple comparisons or tests with a null hypothesis
What is the hidden data problem?
We dont have all the data available to measure the size or effect in reality.
What is the reproducibility problem?
The prevalence of false positives in the literature is higher than we might expect
(reproducibility crisis)
What is a two-tailed test?
The 5% cutoff (alpha) is split evenly across the two ends (“tails”) of the null distribution. Each tail containing 2.5% of the area under the curve
When should we use a two-tailed test?
When we don’t have a strong theoretical explanation about the direction of an effect and are unsure which way it could go.
What is a one-tailed test?
The 5% rejection region is concentrated at one end or “tail” of the null distribution - one tail contains 5% of the total area under the curve.
When would we use a one-tailed test?
We a strong reason to believe that the relationship is only one way.
If the results are the other way, the outcome is meaningless.
What happens if researchers change from a one-tailed to a two-tailed test after looking at the data?
What is this called?
You increase the alpha level from 5% to 7.5%.
We have conducted 2 tests (first test 5% alpha, 1 tail), if we test again again for a 2 tailed we add another 2.5% on the back end and the alpha is now 7.5%
This is called the problem of multiplicity
What is the hocus pocus trick?
When you conduct many experiments where the majority fail, but then due to the 5% alpha a few succeed and you only reveal those ones.
What is the family wise error rate?
Tests that are used to take the error rate of all tests into account to avoid the problem of multilicity.
What is the name of the most popular and conservative family wise error rate calculation?
Bonferroni method
When would we conduct the Bonferroni method during an experiment
Post-hoc
We do the adjustment after the tests
What are the 4 steps for dealing with multiplicity?
- Clearly define the null hypothesis.
- Focus analyses on the most important comparisons.
- Treat post-hoc comparisons as an exploratory analysis.
* Dealing with Multiplicity* - Run follow-up confirmatory experiments with planned contrasts.
What is selection bias?
when the outcome of an experiment or research study influences the decision to publish it.
only publishing positive results and not negative ones
What is the file drawer problem?
That journals are filled with the 5% of studies that show Type 1 errors, while the drawers back at home are filled with the 95% of studies that show nonsignificant results
Selection Bias
What is Inflation Bias (“p-Hacking”)
when researchers try out several statistical analyses and selectively report those that produce significant results.
What are 5 ways you can p-hack your results?
1) stopping data collection after finding a significant p-value
2) recording many DV’s and only reporting DV’s that yield a significant p-value
3) including or dropping outliers to yield a significant p-value
4) excluding, combining, or splitting conditions to yield a significant p-value
5) including or excluding covariates until yielding a significant p-value
What is the texas sharpshooter fallacy
If you are making your hypothesis after finding your results
(an example of p-hacking or HARKING)
Explain the hidden data problem
We ideally would like to analyse all the data (left image) but in reality we only have access to a limited scope of data (right image) and have to infer our results
What are the two classifications of missing data?
Missing at random (values in data set are missing at random)
Missing not at random (value of the variable thats missing is related to the reason its missing)
What is the most problematic type of missing data?
Missing not at random
What are the two ways to deal with missing data?
Imputation (missing data is “filled in”, imputed or replaced with substituted values)
Complete case deletion (all cases with a missing value are deleted)
What is the ceiling/floor effect?
When an independent variable no longer has an effect on a dependent variable, or the level above/below which variance in an independent variable is no longer measurable.
When does the ceiling and floor effect occur?
Ceiling = test is too easy, everyone can get maximum result
Floor = test is too hard, nobody can get off the bottom result
What is a cliff in a data set?
What are the two types of replication?
Direct replication: Experimenter directly replicates the experiment to see if they get the same results
Conceptual replication: Experimenter tries to reproduce findings using a different set of methods that test the same idea.
What are the 3 different types of reproducibility?
Methods reproducibility:
provides sufficient detail about procedures and data so that the same procedures can be exactly repeated.
Results reproducibility:
obtain the same results from an independent study with procedures as closely matched to the original study as possible.
Inferential reproducibility:
draw the same conclusions from either an independent replication of a study or a reanalysis of the original study.
Incentives have been argued to be a reason for the lack of reproducibility and replication studies.
What incentives may lead to a lack of replication studies?
- Hard to publish replications
- Replications are not ‘novel’
- Replications can be costly ($$)
- People don’t like to be wrong
- Replication is hard.
The ____ the sample size, the more likely it will be reproducible
Larger
What are the four scientific principles of mertonian norms that were argued are needed for proper science?
Communalism
Universalism
Disinterestedness
Organised skepticism
