L6 Flashcards

1
Q

What does this show us?

A

Statistical power in signal detection terms

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

What is the multiplicity problem?

A

Running multiple tests of the same null hypothesis can lead to more errors

Dealing with multiple comparisons or tests with a null hypothesis

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

What is the hidden data problem?

A

We dont have all the data available to measure the size or effect in reality.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

What is the reproducibility problem?

A

The prevalence of false positives in the literature is higher than we might expect

(reproducibility crisis)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

What is a two-tailed test?

A

The 5% cutoff (alpha) is split evenly across the two ends (“tails”) of the null distribution. Each tail containing 2.5% of the area under the curve

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

When should we use a two-tailed test?

A

When we don’t have a strong theoretical explanation about the direction of an effect and are unsure which way it could go.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

What is a one-tailed test?

A

The 5% rejection region is concentrated at one end or “tail” of the null distribution - one tail contains 5% of the total area under the curve.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

When would we use a one-tailed test?

A

We a strong reason to believe that the relationship is only one way.

If the results are the other way, the outcome is meaningless.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

What happens if researchers change from a one-tailed to a two-tailed test after looking at the data?

What is this called?

A

You increase the alpha level from 5% to 7.5%.

We have conducted 2 tests (first test 5% alpha, 1 tail), if we test again again for a 2 tailed we add another 2.5% on the back end and the alpha is now 7.5%

This is called the problem of multiplicity

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

What is the hocus pocus trick?

A

When you conduct many experiments where the majority fail, but then due to the 5% alpha a few succeed and you only reveal those ones.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

What is the family wise error rate?

A

Tests that are used to take the error rate of all tests into account to avoid the problem of multilicity.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

What is the name of the most popular and conservative family wise error rate calculation?

A

Bonferroni method

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

When would we conduct the Bonferroni method during an experiment

A

Post-hoc

We do the adjustment after the tests

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

What are the 4 steps for dealing with multiplicity?

A
  1. Clearly define the null hypothesis.
  2. Focus analyses on the most important comparisons.
  3. Treat post-hoc comparisons as an exploratory analysis.
    * Dealing with Multiplicity*
  4. Run follow-up confirmatory experiments with planned contrasts.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

What is selection bias?

A

when the outcome of an experiment or research study influences the decision to publish it.

only publishing positive results and not negative ones

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

What is the file drawer problem?

A

That journals are filled with the 5% of studies that show Type 1 errors, while the drawers back at home are filled with the 95% of studies that show nonsignificant results

Selection Bias

17
Q

What is Inflation Bias (“p-Hacking”)

A

when researchers try out several statistical analyses and selectively report those that produce significant results.

18
Q

What are 5 ways you can p-hack your results?

A

1) stopping data collection after finding a significant p-value
2) recording many DV’s and only reporting DV’s that yield a significant p-value
3) including or dropping outliers to yield a significant p-value
4) excluding, combining, or splitting conditions to yield a significant p-value
5) including or excluding covariates until yielding a significant p-value

19
Q

What is the texas sharpshooter fallacy

A

If you are making your hypothesis after finding your results

(an example of p-hacking or HARKING)

20
Q

Explain the hidden data problem

A

We ideally would like to analyse all the data (left image) but in reality we only have access to a limited scope of data (right image) and have to infer our results

21
Q

What are the two classifications of missing data?

A

Missing at random (values in data set are missing at random)

Missing not at random (value of the variable thats missing is related to the reason its missing)

22
Q

What is the most problematic type of missing data?

A

Missing not at random

23
Q

What are the two ways to deal with missing data?

A

Imputation (missing data is “filled in”, imputed or replaced with substituted values)

Complete case deletion (all cases with a missing value are deleted)

24
Q

What is the ceiling/floor effect?

A

When an independent variable no longer has an effect on a dependent variable, or the level above/below which variance in an independent variable is no longer measurable.

25
When does the ceiling and floor effect occur?
Ceiling = test is too easy, everyone can get maximum result Floor = test is too hard, nobody can get off the bottom result
26
What is a **cliff** in a data set?
27
What are the two types of replication?
**Direct replication:** Experimenter directly replicates the experiment to see if they get the same results **Conceptual replication:** Experimenter tries to reproduce findings using a different set of methods that test the same idea.
28
What are the 3 different types of reproducibility?
**Methods reproducibility**: provides sufficient detail about procedures and data so that the same procedures can be exactly repeated. **Results reproducibility**: obtain the same results from an independent study with procedures as closely matched to the original study as possible. **Inferential reproducibility**: draw the same conclusions from either an independent replication of a study or a reanalysis of the original study.
29
Incentives have been argued to be a reason for the lack of reproducibility and replication studies. What incentives may lead to a lack of replication studies?
1. Hard to publish replications 2. Replications are not ‘novel’ 3. Replications can be costly ($$) 4. People don’t like to be wrong 5. Replication is hard.
30
The ____ the sample size, the more likely it will be reproducible
Larger
31
What are the four scientific principles of **mertonian norms** that were argued are needed for proper science?
Communalism Universalism Disinterestedness Organised skepticism