W11 - P-Values and Multiple NHSTs

Question 1

In theory, what is the number of false rejections we can make?

Answer 1

• Can only incorrectly reject one possible value
• When it is the true population parameter value or equivalently, the true value for the null hypothesis
• All other possible null hypothesised values are false, and hence, correct to reject

Question 2

What are the 2 ways we can run multiple NHSTs

Answer 2

Same variable measured in different samples

• In each sample, one NHST is conducted

Different variables measured in same sample

• In each one variable, one NHST is conduted

Question 3

If we ran multiple NHSTs independently (purely independent), what is the exact probability for at least one false rejection error in all NHSTs?

Answer 3

1 - (1 - α)^k

• α = Alpha Level
• k = Number of NHST tests
• Worst case scenerio

Question 4

How is number of NSHTs related to false rejection error

Answer 4

As number of NHSTs increases, probability of at least one false rejection error increases dramatically

• Negative Acceleration
• By the time we have >25 NHSTs (and H₀ = True), there is a very high probability of rejecting one NHSTs purely be chance alone

Question 5

What is the probability of multiple confidence intervals caputring the true population values?

Answer 5

As number of confidence intervals increases…

• Coverage rate for capturing all true population parameter values smaller than nominal coverage rate
• Higher chance at least one interval will not capture the true population paramter value at the nominal rate

Question 6

What is the example of dependent NHST

Answer 6

Non-Orthogonal Contrasts

Question 7

If NHSTs and CIs are not independent, and we undertake multiple NHSTs and CIs, what will happen to probability of false rejection

Answer 7

Probability of at least one false rejection error is much smaller than expected value of independence of NHSTs.

Question 8

Hence, what factors affects probability of false rejection / coverage in mulitple NHSTs

Answer 8

When undertaking multiple NHSTS,

• The chance of making at least one false rejection error will be higher than the nominal value set for alpha for each NHST if the null hypothesis is true in every NHST
• How much higher will be conditional on how strongly the tests are dependent.
  
  • The more dependent the tests, the smaller the probability of at least one false rejection over multiple tests

Likewise, the probability of all intervals capturing the true population parameter value will be lower than the nominal level set for each interval.

Question 9

What is the curse of mulitiplicity

Answer 9

Possible inflation in false rejection error rate (or type 1 error rate) as we undertake two or more NHSTs or CIs

Question 10

What are the effects of undertaking multiple NHSTS

Answer 10

• Chance of making at least one false rjejection error will be higher than nominal value set to alpha for each NHST (if the null is true)
  
  • How much higher than nominal depends on whether NHSTs are independent/dependent

Question 11

What are the effects of undertaking multiple CIs

Answer 11

• Probability of all intervals capturing the true population parameter value will be lower than nominal level set for each interval

Question 12

What is bonferroni correction

Answer 12

Basis of comparision: Familywise alpha value

• Familywise alpha value divided by no. of NHSTs
• to obtain per comparision alpha value

Question 13

What is familywise alpha value

Answer 13

Value to control the possible false rejection error over ALL NHSTs.

• Control at the Familywise level

Question 14

What is per comparision alpha value

Answer 14

Value assigned to alpha for EACH NHST

• Stricter value

Question 15

Is boneforroni the most severe correction?

Answer 15

It is the most conservative approach possible

Question 16

What are some situations people claim that correction is okay

Answer 16

When the research is a “fishing expedition” with no theory or previous research to guide the research questions being asked

• Hence, be more careful

Question 17

Do we always adjust mulitple NHSTs using bonferroni

Answer 17

Not always. It depends on the rationale (Guided by previous findings, likely sample size, theory, stage of development of research, etc)

Question 18

What are the two things we can do to edit the bonferroni correction

Answer 18

1. Have familywise alpha value >.05 if we wish
2. Not have per comparision alpha values be the same (e.g. .03,.01), as long as they sum to desired familywise alpha value
   
   • Maybe more likely .03
   • Less likely .01

As long as we justify why we do something like this.