Data analysis: Multiple Testing and Analysis of Variance Flashcards
What do we want to measure from a quantitative experiment?
We want to measure the size of an effect
What do random error/random variation cause?
Random error/random variation causes uncertainty in the result
What does taking the mean of results do but not eliminate?
Taking the mean of results of multiple observations decreases uncertainty but doesn’t eliminate uncertainty
What is a measure of uncertainty?
S.e.m is one measure of uncertainty
What C.I is a region +/- 1 s.e.m around the mean and what does this imply?
A region +/- 1 s.e.m around the mean is a 68% confidence interval
This implies that there is a 68% chance that the real answer is within that interval
What C.I is a region +/- 2 s.e.m around the mean?
A region +/- 2 s.e.m around the mean is a 95% confidence interval
What does s.e.m have a problem with and why?
s.e.m has a problem with small numbers of observations because it underestimates MOE
What do you have to do for triplicate observations in order to get a 68% C.I?
For triplicate observations you need to double the s.e.m error bars to get a 68% C.I
What types of experiments are there in ANOVA?
- Independent Measures
- Repeated measures
- Paired data
- Multiple repeated measures
When does paired data typically arise?
Paired data typically arises when you can make measurements on the same subject before and after treatment
What are typical examples of independent measures involving 2 groups?
- Patient vs controls
- Transgenic animals vs control animals
What is the best estimate of the effect size in the 2 groups in an independent measure?
The best estimate of the effect size is the difference between the means of the 2 groups
How can uncertainty be estimated in an independent measure experiment involving 2 groups or more exactly?
The uncertainty can be estimated roughly by looking at s.e.m error bars or more exactly by calculating a 95% CI for the difference in the means using a t-test
What do we assume in order for comparing error bars to work?
- The groups are independent measures, not paired data
- There are at least 10 observations
- The data is roughly similar to a normal distribution
What does comparing error bars give in terms of CI?
Comparing error bars gives a rough estimate of the 80% CI
What does the upper end of the CI tell us?
The upper end of the CI tells you how big the real effect might perhaps be
-Possibly bigger and more biologically significant than the difference in the means
What does the lower end of the CI tell us?
The lower end of the CI tells . you how small the real effect might perhaps be
-Possibly too small to be biologically significant. Possibly zero or even negative
What is a typical example of paired data?
Typical example is when you measure each subject twice, once before and once after treatment
What does paired data eliminate?
Eliminates uncertainty due to variation between individual subjects
How do you estimate effect size of paired data?
- Calculate effect size(before or after) for each individual
- Then take the mean
How do you test uncertainty of paired data?
-Take the list of effect sizes for each individual
-Calculate the s.e.m for this data
-Plot error bars of +/- 2 s.e.m to estimate the 95% CI
-Works reasonable for>10
observations
What’s a better method to test uncertainty of paired data?
-Use a paired t-test to calculate the 95% CI for the effect using a paired t-test
What assumption must we presume in independent measure experiments with multiple groups, in order to calculate the 95% CI and a p . value using a t test?
Assumptions:
- There are a fairly small number of different groups
- It is clear in advance what the experiment question is and what the key comparison is
What do the old statistics and p values in multiple testing tell us?
p<0.05 tells us the chance of getting data that looks like a real effect, when there is no real effect and only random variation
How do we deal with multiple testing?
- Report all comparisons and statistical test
- Use a multiple testing correction
- Bonferroni correction-Multiply the p-value by the number of independent tests
- Conservative test . helps to avoid type 1 error but may cause a lot of type 2 errors - Use a 1-way analysis of variance(ANOVA)
What does 1-way ANOVA look at?
-Looks at all of the data in all of the groups together
-Looks at the overall variation within the groups
-This measures the overall
variability of the data
-Then looks at the overall variation between the groups
What does it mean if the p-value is large in a 1-way ANOVA?
If the p-value is large, it is quite possible that the differences between groups are just random variation
What does it mean if the p-value is small in a 1-way ANOVA?
If the p-value is small, it is less likely that these are random differences
What does it mean if p<0.05 or p<0.005 in a 1-way ANOVA?
If p<0.05 or p<0.005 then the data is statistically significant
What should we do post hoc tests?
-First look at data
-Then confirm with a post hoc test
-This compares 2 groups at a
time
-Gives an individual
conference interval and p-
value for the comparison
-But includes a multiple
testing correction
