Hypothesis testing

Question 1

What is a type I error?

Answer 1

The probability that we will reject the null hypothesis (concluding that there is an effect) when there actually isn’t

A false positive

Thinking you have found gold but it is actually an aluminium can

Question 2

What is the typical level of type I errors in biological sciences?

Answer 2

0.05 / 5%

Question 3

What is a type II error?

Answer 3

Failing to reject a false null hypothesis (when there is an effect)

A false negative

A metal detector doesn’t beep when there is gold underground

Question 4

Test statistic for one-sample t-test

Answer 4

t = difference between measured mean and hypothesised value of the mean / SE of the mean

Question 5

What does a one-sample t-test do?

Answer 5

Tests whether the mean of the sample of data is equal to a hypothesised mean

Question 6

Paired t-test

Answer 6

Special case of the one-sample t-test
We have a pair of measurements per subject (before and after for example)

Question 7

What does a paired t-test do?

Answer 7

Tests whether the mean of the set of differences is equal to zero

Question 8

Two-sample t-test test statistic

Answer 8

t = difference between out measured difference between 2 means and the difference between 2 means that we expect under the null hypothesis / SE of the difference between 2 means

Question 9

What does the two-sample t-test do?

Answer 9

Tests if the means of two samples of data are the same or significantly different

Question 10

Two-sample t-tests assuming equal variances are robust for sample sizes…

Answer 10

> = 30
Per group

Question 11

If the two groups have _____ in each group, then it is still OK to use the two-sample t-test when the SDs of the groups differ by up to ______ fold

Answer 11

Similar numbers
Three

Question 12

What is the t-test where equal variances are not assumed?

Answer 12

Welch’s T-test

Question 13

What is the test statistic for an ANOVA?

Answer 13

F = treatment mean square / error mean square

(explained variance (signal) / unexplained variance (noise))

Question 14

What does an ANOVA do?

Answer 14

Tests if all the means of more than two groups are the same or significantly differ

Question 15

What do hypothesis tests involve?

Answer 15

A test statistic
A distribution from which we expect the test statistic to come from if the null hypothesis is true
Calculation of a p-value

Question 16

What does a p-value tell us?

Answer 16

The probability of obtaining the value of the test statistic, or a more extreme value, if the null hypothesis is true

Question 17

What is an interaction?

Answer 17

When the effect of one variable on the outcome variable differs depending on the level of another variable

The effect of a drug on blood pressure differs depending on the type of diet they eat

Question 18

Assumptions of a t-test

Answer 18

Data are normally distributed

Question 19

Assumptions of ANOVA

Answer 19

The error (residuals) is normally distributed
Variance of the unexplained variation is constant throughout the dataset (homogeneity of error/residuals)

Question 20

Chi Squared Test statistic

Answer 20

χ2 = sum( (observed - expected)2 / expected )

Question 21

How do you calculate the expected value in Chi Squared?

Answer 21

Expected = (row total x column total) / grand total

Question 22

Assumptions of a Chi Squared Test

Answer 22

All expected counts are >= 5
Mean of counts = the variance of the counts
The Poisson distribution can be approximated by a Normal distribution, which is true as long as the mean is >= 5

Question 23

If all counts aren’t >= 5 in Chi Squared Test, when is it still reasonable to run the test?

Answer 23

As long as all expected counts are >= 1
80% of the expected counts are >= 5

Question 24

How to check for normally distributed residuals?

Answer 24

Plot histogram of the residuals and check the shape

Plot the normal Q-Q plot of the residuals (the dots should be not too far from the line, shouldn’t bend away from the line in any systematic fashion)

Question 25

How to check the homogeneity of residuals/error?

Answer 25

Residuals vs fitted values plot

Looking to see that there are no obvious differences between the groups, (they don’t get bigger as the fitted values get bigger for example) scatter of positive and negative residuals in each case

Question 26

Bonferroni correction

Answer 26

overall p-value / number of tests conducted

This gives a new p-value that you can use as a threshold

Question 27

Why do we correct our p-value for multiple comparisons?

Answer 27

When we do a statistical test, there is a 5% chance that we will reject a true null hypothesis (get a false positive)
If we conduct multiple analyses, there is a 5% chance each time that this will happen, so our error gets bigger (for 3 tests there is 5 + 5 + 5% = 15% chance that we will get a false positive)
We need to correct for this so that our results still only have 5% error overall

Question 28

What can we do if our data don’t fit the assumptions of a statistical test?

Answer 28

Transform the data

Question 29

What types of transformation can we do?

Answer 29

Log the data
Square the data
Square root the data

Question 30

Non-parametric alternative to two-sample t-test

Answer 30

Mann-Whitney U Test

Question 31

Null hypothesis of Mann-Whitney U Test

Answer 31

The two groups being compared come from the same distribution, with the same median

It doesn’t matter what the shape of that distribution is

Question 32

How to conduct a Mann-Whitney U Test

Answer 32

1. Convert the values into ranks
2. Find the sum of ranks in both samples
3. Calculate U for both samples
4. Test statistic (U) = minimum of U1 and U2

Question 33

Assumptions of the Mann-Whitney U Test

Answer 33

The two groups being compared follow the same distribution (if one is left-skewed, the other group should be left skewed)