Chapters 18-20 Flashcards
What does it mean if the results are statistically significant?
Means that the results would be surprising, but not impossible if the null hypothesis were true. But we really care about effect size
The relationship between the P value and size of the effect depends on…
sample size
Small effect can be significant with…
a large sample.
Relative risk
The ratio of the treated group to control group (larger means greater effect size)
False Discovery Rate (FDR)
Given that the result is significant, the probability that the null hypothesis is actually true. This cannot be truly known.
The FDR probability cannot usually be determined because…
every probability calculation must be based on a model that can be expressed mathematically.
FDR=
Type I Errors/Total Decisions to Reject Null
However, our interpretation always involved at least an informal process of trying to estimate the FDR.
Examples 1, 2, and 3
Example 1 of informally estimating the FDR:
P value is
Example 2 of informally estimating the FDR:
P value is
Example 3 of informally estimating the FDR:
P value is P value is
What if you somehow know the probability that the null hypothesis is false?
Such knowledge would be an example of a prior probability
Perhaps a prior probability could be based on…
your previous experiences (e.g. 50% of people brought to trial int he past really were criminals)
Power
The power of a statistical test is the probability that it correctly rejects the null hypothesis when the null is false
Power can be calculated if you know:
1) sample size 2) standard deviation 3) size of effect of interest
Power=
Decisions to reject null/# times null is false
FDR=
type I errors/total # decisions to reject null
The Bayesian approach to FDR
Can have a big impact on interpretation! • Example 1: You have a drug developer who has a track record of 80%. • Example 2: You have experience with traditional herbal remedies producing desired effect level 10% of the time.
The Bayesian Approach to FDR (2)
Example 1: • FDR = • Example 2: • FDR = Effect of Prior Probability (0.10) on Interpretation • In drug trials with Effect of Prior Probability (0.80) on Interpretation the same level of statistical power, • Example 1: • FDR = 10/650 = 0.15 • Example 2: • FDR = 45/125 = 0.36
FDR:
proportion of significant results that are really type I errors
The Bayesian Approach to FDR is…
no more accurate than your prior probabilities
Not statistically significant means…
only that the P value is larger than the present threshold. Does not prove that the null hypothesis is true.
Keep in mind, the relationship between the P value and
size of the effect depends on sample size
Large effect can be nonsignificant with a small sample.
CI helps put the data in perspective
- Testing whether the amount of epinephrine receptors on cells is different in people with high blood pressure. - Used platelets as a convenient ‘cell’ type to measure. - Means were similar. - P value = 0.81 If you stop at this point and publish only these results, you might imply that the null is true.
Example 1:
- Means were similar.
- P value = 0.81
- The observed means only differ by +6. - Out of total mean of 260. - But 95% CI for the difference is -45 to +57. - Approximately ±20%. - Would it matter if the true difference is actually 20%? -You need to decide whether this level of uncertainty matters in the scientific/medical context
Example 2:
A large study tests the impacts of sonograms. -They are expensive but common. Two treatments: 1) 2 routine sonograms 2) 0 unless physician saw a special need Observations: -count adverse health outcomes for fetus. Results pretty similar (P=0.86) with slightly higher adverse outcomes when sonograms used more often.
Outcomes similar: - P value = 0.86 - Null: relative risk = 1.0 - 95% CI for relative risk: - 0.88 to 1.17 - So it is possible that sonograms reduce risk by 12% or raise risk by 17%. - But we definitely can’t rule out the null hypothesis.
So are they useful? What if the null was true (no
impact on safety)? Might they still be ‘useful’?
How to Get Narrower CIs
. • Unless you can change the SD (not usually), then sample size must be increased. • Rule of thumb: - Increase sample size by factor of 4 to decrease CI by factor of 2.
Recall: the width of the CI is inversely proportional
to
the square root of the sample size.
If you find that the CI contains some
unacceptable uncertainty, then you
need to
run a new study with greater
sample size to narrow the CI .
The power of a
statistical test is
the probability that it correctly rejects the null hypothesis when the null is false.
`As power increases, the probability of
Type II errors
decreases
Given the 1) sample size, 2) standard deviation, and
3) size of effect, power is
the fraction of experiments
in which you would expect to find a significant result.
• In practice, power
is the
fraction of experiments in which you would expect to find a statistically significant result.
Which means that power can never be lower than α.
As effect size decreases to zero, power decreases to α.
When effect size = 0, then null is true, and the only
way to get a significant result is by chance (sampling error).
