Data analysis: Multiple Testing and Analysis of Variance

Question 1

What do we want to measure from a quantitative experiment?

Answer 1

We want to measure the size of an effect

Question 2

What do random error/random variation cause?

Answer 2

Random error/random variation causes uncertainty in the result

Question 3

What does taking the mean of results do but not eliminate?

Answer 3

Taking the mean of results of multiple observations decreases uncertainty but doesn’t eliminate uncertainty

Question 4

What is a measure of uncertainty?

Answer 4

S.e.m is one measure of uncertainty

Question 5

What C.I is a region +/- 1 s.e.m around the mean and what does this imply?

Answer 5

A region +/- 1 s.e.m around the mean is a 68% confidence interval
This implies that there is a 68% chance that the real answer is within that interval

Question 6

What C.I is a region +/- 2 s.e.m around the mean?

Answer 6

A region +/- 2 s.e.m around the mean is a 95% confidence interval

Question 7

What does s.e.m have a problem with and why?

Answer 7

s.e.m has a problem with small numbers of observations because it underestimates MOE

Question 8

What do you have to do for triplicate observations in order to get a 68% C.I?

Answer 8

For triplicate observations you need to double the s.e.m error bars to get a 68% C.I

Question 9

What types of experiments are there in ANOVA?

Answer 9

1. Independent Measures
2. Repeated measures
   - Paired data
   - Multiple repeated measures

Question 10

When does paired data typically arise?

Answer 10

Paired data typically arises when you can make measurements on the same subject before and after treatment

Question 11

What are typical examples of independent measures involving 2 groups?

Answer 11

• Patient vs controls

- Transgenic animals vs control animals

Question 12

What is the best estimate of the effect size in the 2 groups in an independent measure?

Answer 12

The best estimate of the effect size is the difference between the means of the 2 groups

Question 13

How can uncertainty be estimated in an independent measure experiment involving 2 groups or more exactly?

Answer 13

The uncertainty can be estimated roughly by looking at s.e.m error bars or more exactly by calculating a 95% CI for the difference in the means using a t-test

Question 14

What do we assume in order for comparing error bars to work?

Answer 14

• The groups are independent measures, not paired data
• There are at least 10 observations
• The data is roughly similar to a normal distribution

Question 15

What does comparing error bars give in terms of CI?

Answer 15

Comparing error bars gives a rough estimate of the 80% CI

Question 16

What does the upper end of the CI tell us?

Answer 16

The upper end of the CI tells you how big the real effect might perhaps be
-Possibly bigger and more biologically significant than the difference in the means

Question 17

What does the lower end of the CI tell us?

Answer 17

The lower end of the CI tells . you how small the real effect might perhaps be
-Possibly too small to be biologically significant. Possibly zero or even negative

Question 18

What is a typical example of paired data?

Answer 18

Typical example is when you measure each subject twice, once before and once after treatment

Question 19

What does paired data eliminate?

Answer 19

Eliminates uncertainty due to variation between individual subjects

Question 20

How do you estimate effect size of paired data?

Answer 20

1. Calculate effect size(before or after) for each individual
2. Then take the mean

Question 21

How do you test uncertainty of paired data?

Answer 21

-Take the list of effect sizes for each individual
-Calculate the s.e.m for this data
-Plot error bars of +/- 2 s.e.m to estimate the 95% CI
-Works reasonable for>10
observations

Question 22

What’s a better method to test uncertainty of paired data?

Answer 22

-Use a paired t-test to calculate the 95% CI for the effect using a paired t-test

Question 23

What assumption must we presume in independent measure experiments with multiple groups, in order to calculate the 95% CI and a p . value using a t test?

Answer 23

Assumptions:

• There are a fairly small number of different groups
• It is clear in advance what the experiment question is and what the key comparison is

Question 24

What do the old statistics and p values in multiple testing tell us?

Answer 24

p<0.05 tells us the chance of getting data that looks like a real effect, when there is no real effect and only random variation

Question 25

How do we deal with multiple testing?

Answer 25

1. Report all comparisons and statistical test
2. Use a multiple testing correction
   - Bonferroni correction-Multiply the p-value by the number of independent tests
   - Conservative test . helps to avoid type 1 error but may cause a lot of type 2 errors
3. Use a 1-way analysis of variance(ANOVA)

Question 26

What does 1-way ANOVA look at?

Answer 26

-Looks at all of the data in all of the groups together
-Looks at the overall variation within the groups
-This measures the overall
variability of the data
-Then looks at the overall variation between the groups

Question 27

What does it mean if the p-value is large in a 1-way ANOVA?

Answer 27

If the p-value is large, it is quite possible that the differences between groups are just random variation

Question 28

What does it mean if the p-value is small in a 1-way ANOVA?

Answer 28

If the p-value is small, it is less likely that these are random differences

Question 29

What does it mean if p<0.05 or p<0.005 in a 1-way ANOVA?

Answer 29

If p<0.05 or p<0.005 then the data is statistically significant

Question 30

What should we do post hoc tests?

Answer 30

-First look at data
-Then confirm with a post hoc test
-This compares 2 groups at a
time
-Gives an individual
conference interval and p-
value for the comparison
-But includes a multiple
testing correction