L6

Question 1

What does this show us?

Answer 1

Statistical power in signal detection terms

Question 2

What is the multiplicity problem?

Answer 2

Running multiple tests of the same null hypothesis can lead to more errors

Dealing with multiple comparisons or tests with a null hypothesis

Question 3

What is the hidden data problem?

Answer 3

We dont have all the data available to measure the size or effect in reality.

Question 4

What is the reproducibility problem?

Answer 4

The prevalence of false positives in the literature is higher than we might expect

(reproducibility crisis)

Question 5

What is a two-tailed test?

Answer 5

The 5% cutoff (alpha) is split evenly across the two ends (“tails”) of the null distribution. Each tail containing 2.5% of the area under the curve

Question 6

When should we use a two-tailed test?

Answer 6

When we don’t have a strong theoretical explanation about the direction of an effect and are unsure which way it could go.

Question 7

What is a one-tailed test?

Answer 7

The 5% rejection region is concentrated at one end or “tail” of the null distribution - one tail contains 5% of the total area under the curve.

Question 8

When would we use a one-tailed test?

Answer 8

We a strong reason to believe that the relationship is only one way.

If the results are the other way, the outcome is meaningless.

Question 9

What happens if researchers change from a one-tailed to a two-tailed test after looking at the data?

What is this called?

Answer 9

You increase the alpha level from 5% to 7.5%.

We have conducted 2 tests (first test 5% alpha, 1 tail), if we test again again for a 2 tailed we add another 2.5% on the back end and the alpha is now 7.5%

This is called the problem of multiplicity

Question 10

What is the hocus pocus trick?

Answer 10

When you conduct many experiments where the majority fail, but then due to the 5% alpha a few succeed and you only reveal those ones.

Question 11

What is the family wise error rate?

Answer 11

Tests that are used to take the error rate of all tests into account to avoid the problem of multilicity.

Question 12

What is the name of the most popular and conservative family wise error rate calculation?

Answer 12

Bonferroni method

Question 13

When would we conduct the Bonferroni method during an experiment

Answer 13

Post-hoc

We do the adjustment after the tests

Question 14

What are the 4 steps for dealing with multiplicity?

Answer 14

1. Clearly define the null hypothesis.
2. Focus analyses on the most important comparisons.
3. Treat post-hoc comparisons as an exploratory analysis.
   * Dealing with Multiplicity*
4. Run follow-up confirmatory experiments with planned contrasts.

Question 15

What is selection bias?

Answer 15

when the outcome of an experiment or research study influences the decision to publish it.

only publishing positive results and not negative ones

Question 16

What is the file drawer problem?

Answer 16

That journals are filled with the 5% of studies that show Type 1 errors, while the drawers back at home are filled with the 95% of studies that show nonsignificant results

Selection Bias

Question 17

What is Inflation Bias (“p-Hacking”)

Answer 17

when researchers try out several statistical analyses and selectively report those that produce significant results.

Question 18

What are 5 ways you can p-hack your results?

Answer 18

1) stopping data collection after finding a significant p-value
2) recording many DV’s and only reporting DV’s that yield a significant p-value
3) including or dropping outliers to yield a significant p-value
4) excluding, combining, or splitting conditions to yield a significant p-value
5) including or excluding covariates until yielding a significant p-value

Question 19

What is the texas sharpshooter fallacy

Answer 19

If you are making your hypothesis after finding your results

(an example of p-hacking or HARKING)

Question 20

Explain the hidden data problem

Answer 20

We ideally would like to analyse all the data (left image) but in reality we only have access to a limited scope of data (right image) and have to infer our results

Question 21

What are the two classifications of missing data?

Answer 21

Missing at random (values in data set are missing at random)

Missing not at random (value of the variable thats missing is related to the reason its missing)

Question 22

What is the most problematic type of missing data?

Answer 22

Missing not at random

Question 23

What are the two ways to deal with missing data?

Answer 23

Imputation (missing data is “filled in”, imputed or replaced with substituted values)

Complete case deletion (all cases with a missing value are deleted)

Question 24

What is the ceiling/floor effect?

Answer 24

When an independent variable no longer has an effect on a dependent variable, or the level above/below which variance in an independent variable is no longer measurable.

Question 25

When does the ceiling and floor effect occur?

Answer 25

Ceiling = test is too easy, everyone can get maximum result

Floor = test is too hard, nobody can get off the bottom result

Question 26

What is a cliff in a data set?

Answer 26

Question 27

What are the two types of replication?

Answer 27

Direct replication: Experimenter directly replicates the experiment to see if they get the same results

Conceptual replication: Experimenter tries to reproduce findings using a different set of methods that test the same idea.

Question 28

What are the 3 different types of reproducibility?

Answer 28

Methods reproducibility:

provides sufficient detail about procedures and data so that the same procedures can be exactly repeated.

Results reproducibility:

obtain the same results from an independent study with procedures as closely matched to the original study as possible.

Inferential reproducibility:

draw the same conclusions from either an independent replication of a study or a reanalysis of the original study.

Question 29

Incentives have been argued to be a reason for the lack of reproducibility and replication studies.

What incentives may lead to a lack of replication studies?

Answer 29

1. Hard to publish replications
2. Replications are not ‘novel’
3. Replications can be costly ($$)
4. People don’t like to be wrong
5. Replication is hard.

Question 30

The ____ the sample size, the more likely it will be reproducible

Answer 30

Larger

Question 31

What are the four scientific principles of mertonian norms that were argued are needed for proper science?

Answer 31

Communalism

Universalism

Disinterestedness

Organised skepticism