Statistical theory 3

Question 1

What is null hypothesis significance testing called for frequentists?

Answer 1

orthodox hypothesis testing

or, null hypothesis significance testing (NHST)

Question 2

What was Fisher’s thoughts on hypothesis testing?

Answer 2

Hypothesis testing is about trying to falsify a single hypothesis

Question 3

What was Neyman’s theory on hypothesis testing?

Answer 3

Hypothesis testing is about choosing between two rival hypotheses

Question 4

What does the null hypothesis predict?

Answer 4

There will be no effect

Question 5

In null hypothesis testing, what is H1?

Answer 5

The alternative hypothesis

This one is looking for some effect (usually the one they want to find)

Question 6

Do we always test the hypothesis or null hypothesis?

Answer 6

Null hypothesis

Question 7

What is a statistical hypothesis?

Answer 7

Claims about population parameters

Question 8

“A critical part of designing a study is being able to specify how your 1)_____ hypothesis (what you’re interested in) maps onto your 2)______ hypothesis (what you can actually test)”

Answer 8

1) Research
2) Statistical

Question 9

How do we evaluate whether the null or alternative hypothesis is accurate?

Answer 9

By determining how likely our data would be if the null is true.

If they are unlikely ‘enough’, we reject the null

Question 10

In hypothesis testing, what is a Type 1 error?

Answer 10

False positive

Question 11

In hypothesis testing, what is a Type 2 error?

Answer 11

False negative

Question 12

What type of error rate does null hypothesis testing control for?

Answer 12

Type 1 error rate

Question 13

Why is null hypothesis significance testing designed the way it is?

Answer 13

To control for belief bias (avoid type 1 errors)

Forces a minimum evidentiary standard on you

Question 14

What is the significance level?

Answer 14

If we would expect to see our data n% of the time (or less) if the null were true, we reject the null

The choice for n is arbitrary and is known as the significance level

Question 15

What do you need to build your own statistical test?

Important for all types of statistical tests

Answer 15

A diagnostic test statistic T

Sampling distribution of T if the null is true

The observed T in your data

A rule that maps every value of T onto a decision (accept or reject H0)

Question 16

What is a diagnostic test statistic (T )?

Answer 16

A single number that you can calculate only from your observations

As long as it’s just one number, it’s a test statistic

Question 17

What are these all examples of?

• The mean of a set of observations*
• The standard deviation of a set of observations*
• The third-largest of a set of observations*
• A number of observations*

Answer 17

Diagnostic test statistics

Question 18

Is the two largest numbers in the sample an example of a test statistic?

Answer 18

No

(multiple numbers)

Question 19

How do you calculate sampling distribution if the null is true?

Answer 19

1st. Assume H0 is true

Figure out what the values of your test statistic you should expect

Question 20

A test statistic is diagnostic if the null hypothesis and alternative predict _____ values

Answer 20

different

• e.g. Baby v cat cuteness: number of baby choices out of 100*
• H0: Should be around 50*
• H1: Should* not be around 50

Question 21

In a binomial test statistic, when should we reject the null?

Answer 21

When 95% of the data accepts the H0 and the results lie outside of that.

Question 22

Where is the ‘rejection region’ for null hypothesis testing?

Answer 22

Outside of the 95% probability range.

Question 23

95% probability means that if the null hypothesis is true, there is a 5% chance of _____ rejecting it.

Answer 23

falsely

Question 24

What is a ‘two-tailed’ test?

Answer 24

A directionless test

When the rejection region for the null hypothesis lies on both sides of the distribution

Question 25

What is a ‘one-tailed’ test?

Answer 25

A directional test

Rejection region only covers one tail (still 5%)

Question 26

Why do we use 5% for a desired Type 1 error rate?

Answer 26

α = .05 is the default significance level that we can use in science, can also be α = .01 or α = .001

Question 27

What is a p-value according to Neyman (null hypothesis guy)?

Answer 27

p describes the Type 1 error rate you must be willing to tolerate if you want to reject H0

p is the smallest significance level your data would let you adopt while still being able to reject the null

Question 28

What is a p-value according to Fisher?

Answer 28

p is the probability - if H0 is true - of observing a test statistic at least as extreme as the one that was actually found

A measure of how implausible the data are, according to H0

Question 29

When running a null-hypothesis hypothesis test, what must we do?

Answer 29

Adopt a ‘standard’ significance level of α = .05

If p < α reject the null hypothesis

Otherwise, accept (or retain) the null hypothesis

Question 30

Is the phrase “p is the probability that the null hypothesis is true” accurate?

Answer 30

No, never say this

• P-value is a claim about how likely you were to see your data if the null hypothesis were true. This is not the same thing as a claim about whether the null hypothesis is true*
• A claim about H0 is true depends on what other hypotheses you’re considering, for that you need to be able to evaluate them.*

Question 31

How do you report a p-value?

Answer 31

Question 32

Which of the following groups of phrases are correct?

1) “The null hypothesis is true”
“The alternative hypothesis is false”
We have “proved” that…

Or

2) “We retain the null hypothesis”
“We failed to reject the null hypothesis”
“The test was not significant”
(“Accept” the null)

Or

3) “We reject the null hypothesis”
“The test was significant”
(“Accept” the alternative)

Answer 32

2 and 3

1 are all very definitive statements, they imply we know the truth but we do not know the truth