Hypothesis Testing- ERRORS

Question 1

If p </= 0.05, would our results be likely or unlikely if the null hypothesis were true?

Answer 1

Unlikely

Question 2

If p </= 0.05 (& our results unlikely if the null hypothesis were true), would we reject or fail to reject the null hypothesis?

Answer 2

We would reject the null hypothesis

Question 3

If p > 0.05, would our results be likely or unlikely if the null hypothesis were true?

Answer 3

Likely

Question 4

If p > 0.05 (& our results unlikely if the null hypothesis were true), would we reject or fail to reject the null hypothesis?

Answer 4

We would fail to reject the null hypothesis

Question 5

Is decision-making based on p-values fallible or infallible?

Answer 5

Fallible

Question 6

What is the consequence of decision-making based on p-values being fallible?

Answer 6

That there are 2 types of errors that we can make when deciding to reject/ fail to reject the null hypothesis.

Question 7

If the reality is that our null hypothesis (based on a population) is true, & our decision (inference based on a sample) is to fail to reject our null hypothesis, would there be an error or would we be correct?

Answer 7

We would be correct

Question 8

If the reality is that our null hypothesis (based on a population) is false, & our decision (inference based on a sample) is to fail to reject our null hypothesis, would there be an error or would we be correct?

Answer 8

There would be an error

Question 9

If the reality is that our null hypothesis (based on a population) is true, & our decision (inference based on a sample) is to reject our null hypothesis, would there be an error or would we be correct?

Answer 9

There would be an error

Question 10

If the reality is that our null hypothesis (based on a population) is false, & our decision (inference based on a sample) is to reject our null hypothesis, would there be an error or would we be correct?

Answer 10

We would be correct

Question 11

What is 1 way in which we can express the null hypothesis?

Answer 11

As H0.

Question 12

What is a type I error?

Answer 12

If the reality were that our null hypothesis (based on a population) is true, & our decision (inference based on a sample) was to reject our null hypothesis (we find effects that don’t exist).

Question 13

What is a type II error?

Answer 13

If the reality were that our null hypothesis (based on a population) is false, & our decision (inference based on a sample) was to fail to reject our null hypothesis (we miss effects that do exist).

Question 14

How is it possible to never confuse type I & II errors again?

Answer 14

Simply by remembering the “Boy Who Cried Wolf” analogy (& replacing the word “wolf” with the word “effect”)

Question 15

In which order did the boy who cried wolf cause type I & type II errors?

Answer 15

He caused type I errors first, then a type II error.

Question 16

What happened in the “Boy Who Cried Wolf” analogy?

Answer 16

First, everyone believed that there was a wolf when there wasn’t, and then everyone believed there was no wolf when there was.

Question 17

What is another name for a type I error?

Answer 17

A false positive

Question 18

How could the “Boy Who Cried Wolf” analogy be applied to type I errors?

Answer 18

The boy would cry wolf, causing the villagers to incorrectly reject the null hypothesis of there not being a wolf.

Question 19

How could the “Boy Who Cried Wolf” analogy be applied to type II errors?

Answer 19

There would be a wolf, however, the villagers would incorrectly accept the null hypothesis of there not being a wolf.

Question 20

In what way is guarding against errors a balancing act?

Answer 20

If we set our statistical significance threshold very low to reduce the chance of type I errors (e.g. p<= 5x10^-8), this increases our chance of making a type II error and missing real effects.

Question 21

What is the 5% significance threshold a tradeoff between in relation to errors?

Answer 21

Type I & type II errors.

Question 22

When can we reject the null hypothesis & say we have a significant effect?

Answer 22

If our probability (p-value) of the effect occurring by chance is less than 0.05.

Question 23

What is the chance of finding an effect at 0.05 if the null hypothesis is true?

Answer 23

Less than 1 in 20.

Question 24

When do we fail to reject the null hypothesis?

Answer 24

If our probability (p-value) of the effect occurring by chance is greater than 0.05.