Lecture 7

Question 1

What are the 3 misconceptions about null hypothesis significance testing?

Answer 1

1) Significant=important
2) “Non-significant”= null hypothesis is true
3) “Significant” = null hypothesis is false

Question 2

What is one of the biggest problems in NHST?

Answer 2

Encourages all-or-nothing thinking. Causes researchers to engage in sketchy practices to find a significant difference. P value is arbitrary

Question 3

What was the reproducibility project?

Answer 3

Looked at 100 psych studies, Follow methods of study exactly to see if it was the same. 97% of studies reported significant results

Question 4

What percentage of studies were actually significant in the reproducibility project?

Answer 4

36%. 61% of studies claimed a significant difference that replication didn’t find.

Question 5

Why do we care about reproducibility?

Answer 5

1) False positives-blur the picture of truth
2) Can lead to horrible public policy decisions (ex: turning right on a red light)
3) Waste of resources in pursuing line of faulty thinking
4) False ideas persist in literature due to the fact that only significant results get published.
5) Loss of credibility as a field

Question 6

What did a meta-analysis show on the actual effects of turning right on a red light?

Answer 6

Original studies showed no difference (due to being underpowered). Meta-analysis showed a 61% increase in hitting pedestrians, and a 100% increase in cyclists.

Question 7

What are incentive structures?

Answer 7

The idea that science is objective, and the search for the truth. Scientists must compete for resources. If you are published more, you get to do more stuff. Journals only publish significant results.

Question 8

What is the publication bias?

Answer 8

90% of studies in psychology are statistically significant.

Question 9

What is the publication bias driven by?

Answer 9

Journal editors/reviewers selecting (7/10 papers are rejected)- researchers also only bother to send in significant results

Question 10

What is researcher degrees of freedom?

Answer 10

Researchers contributing to the problem through the choices they make. Abuse of freedom.

Question 11

What is selective reporting?

Answer 11

When the researcher reports only significant measures, even though they measured more than that, without correcting for multiple tests. Type 1 error gets inflated due to this.

Question 12

How do you calculate an increase in type 1 error due to multiple groups?

Answer 12

Do 1- (probability of none) exponent is the number of groups you’re measuring

Question 13

How do researchers deal with extremes?

Answer 13

They choose which scores are extremme and which aren’t. There are also different methods of dealing with outliers such as data transformation and deletion.

Question 14

What percentage of researchers admit to stopping data collection once they achieve significance?

Answer 14

70%

Question 15

What can stopping data collection lead to?

Answer 15

Truth inflation/Magnitude error

Question 16

What is the Truth inflation/magnitude error

Answer 16

Small sample size can lead to an exaggerated effect of significance-significant results are usually not found with small sample sizes except out of sheer luck. Can lead to study getting published based solely on luck, and not based on the actual averages achieved. Be suspicious of studies with huge effects and small sample sizes.