Research Skills 9 : Statistical significance

Question 1

What is Significance?

Answer 1

1. Sufficiently great or important to be worthy of attention

2. Having a particular meaning; signifying something; indicative of something.

Question 2

What does Sufficiently great or important to be worthy of attention mean?

Answer 2

This is a statement about how large the effect is and if it is biologically significant

it is not a statement about statistical significance!!

Question 3

What is Statistical Significance?

Answer 3

1. Choose a number- any number (most people choose 0.05)
2. Call this number α
3. Perform a statistical test
4. If p ≤ α you declare that the result is “statistically significant”
5. If p > α you declare the result is “not statistically significant”

Question 4

What are the two different ways of using p values?

Answer 4

1. Use the actual p-value as a measure of statistical strength
   - This is the method we have been using so far
2. Choose an arbitrary α and use this to declare the result “statistically significant” or “not statistically significant”

Question 5

Method 2 of P values

Answer 5

• If p ≤ α you must reject the null hypothesis

- If p > α you must accept the null hypothesis

Question 6

What is type 1 error ?

Answer 6

Rejecting the null hypothesis, when in fact there is no real effect

Question 7

Why method 2 is flawed?

Answer 7

• When we use 0.05 as a cut off, the Type 1 error rate may be rather high
• If we use 0.01 as our cut off, this is a better bet, but a Type 1 error rate around 20% is still quite possible

Question 8

What is type 2 error?

Answer 8

If we accept the null hypothesis, and declare that a result is “not statistically significant” when, in fact, there is a real effect

Question 9

Why would type 2 error be high?

Answer 9

It depends on the size of the experiment

statisticians call the STATISTICAL POWER

Question 10

What can you also look at to give more information?

Answer 10

Error bars

Question 11

Why is null hypothesis significance testing popular?

Answer 11

• Appears to remove uncertainty- it gives a definite answer
• Significant or not significant
• Reject null hypothesis/ accept null hypothesis
• Lets the computer make the decision for you
• Tells you that your result is significant, so you don’t have to do any more experiments
• Looks like it is based on mathematical logic (it isn’t)
• Helps you get the most published papers for the minimum work

Question 12

How to interpret statistical significance?

Answer 12

1. Always look at the effect size first
   - Statistical significance does not tell you the biological or clinical significance
2. Always use the p-value in combination with s.e.m. error bars or the 95% C.I.
3. Look at the actual p-value to judge the statistical strength of the data

Question 13

How to interpret statistical significance? ctnd

Answer 13

4. Statistically significant” means that your data does not look a lot like random data
5. “Not statistically significant” could mean either
   There may be a real effect, but your data is not strong enough to prove this, or
   There is no effect, or only a very small effect
   Error bars and C.I. can help you distinguish these possibilities