Chapters 18-20

Question 1

What does it mean if the results are statistically significant?

Answer 1

Means that the results would be surprising, but not impossible if the null hypothesis were true. But we really care about effect size

Question 2

The relationship between the P value and size of the effect depends on…

Answer 2

sample size

Question 3

Small effect can be significant with…

Answer 3

a large sample.

Question 4

Relative risk

Answer 4

The ratio of the treated group to control group (larger means greater effect size)

Question 5

False Discovery Rate (FDR)

Answer 5

Given that the result is significant, the probability that the null hypothesis is actually true. This cannot be truly known.

Question 6

The FDR probability cannot usually be determined because…

Answer 6

every probability calculation must be based on a model that can be expressed mathematically.

Question 7

FDR=

Answer 7

Type I Errors/Total Decisions to Reject Null

Question 8

However, our interpretation always involved at least an informal process of trying to estimate the FDR.

Answer 8

Examples 1, 2, and 3

Question 9

Example 1 of informally estimating the FDR:

Answer 9

P value is

Question 10

Example 2 of informally estimating the FDR:

Answer 10

P value is

Question 11

Example 3 of informally estimating the FDR:

Answer 11

P value is P value is

Question 12

What if you somehow know the probability that the null hypothesis is false?

Answer 12

Such knowledge would be an example of a prior probability

Question 13

Perhaps a prior probability could be based on…

Answer 13

your previous experiences (e.g. 50% of people brought to trial int he past really were criminals)

Question 14

Power

Answer 14

The power of a statistical test is the probability that it correctly rejects the null hypothesis when the null is false

Question 15

Power can be calculated if you know:

Answer 15

1) sample size 2) standard deviation 3) size of effect of interest

Question 16

Power=

Answer 16

Decisions to reject null/# times null is false

Question 17

FDR=

Answer 17

type I errors/total # decisions to reject null

Question 18

The Bayesian approach to FDR

Answer 18

Can have a big impact on interpretation!
• Example 1: You have
a drug developer
who has a track
record of 80%.
• Example 2: You have
experience with
traditional herbal
remedies producing
desired effect level
10% of the time.

Question 19

The Bayesian Approach to FDR (2)

Answer 19

Example 1:
• FDR =
• Example 2:
• FDR =
Effect of Prior Probability (0.10) on Interpretation
• In drug trials with Effect of Prior Probability (0.80) on Interpretation
the same level of
statistical power,
• Example 1:
• FDR = 10/650 = 0.15
• Example 2:
• FDR = 45/125 = 0.36

Question 20

FDR:

Answer 20

proportion of significant results that are really type I errors

Question 21

The Bayesian Approach to FDR is…

Answer 21

no more accurate than your prior probabilities

Question 22

Not statistically significant means…

Answer 22

only that the P value is larger than the present threshold. Does not prove that the null hypothesis is true.

Question 23

Keep in mind, the relationship between the P value and

size of the effect depends on sample size

Answer 23

Large effect can be nonsignificant with a small sample.

Question 24

CI helps put the data in perspective

Answer 24

- Testing whether the
amount of epinephrine
receptors on cells is
different in people with
high blood pressure.
- Used platelets as a
convenient ‘cell’ type
to measure.
- Means were similar.
- P value = 0.81
If you stop at this point and publish only these
results, you might imply that the null is true.

Question 25

Example 1: - Means were similar. - P value = 0.81

Answer 25

``` - The observed means only differ by +6. - Out of total mean of 260. - But 95% CI for the difference is -45 to +57. - Approximately ±20%. - Would it matter if the true difference is actually 20%? -You need to decide whether this level of uncertainty matters in the scientific/medical context ```

Question 26

Example 2:

Answer 26

``` A large study tests the impacts of sonograms. -They are expensive but common. Two treatments: 1) 2 routine sonograms 2) 0 unless physician saw a special need Observations: -count adverse health outcomes for fetus. Results pretty similar (P=0.86) with slightly higher adverse outcomes when sonograms used more often. ```

Question 27

``` Outcomes similar: - P value = 0.86 - Null: relative risk = 1.0 - 95% CI for relative risk: - 0.88 to 1.17 - So it is possible that sonograms reduce risk by 12% or raise risk by 17%. - But we definitely can’t rule out the null hypothesis. ```

Answer 27

So are they useful? What if the null was true (no | impact on safety)? Might they still be ‘useful’?

Question 28

How to Get Narrower CIs

Answer 28

``` . • Unless you can change the SD (not usually), then sample size must be increased. • Rule of thumb: - Increase sample size by factor of 4 to decrease CI by factor of 2. ```

Question 29

Recall: the width of the CI is inversely proportional

Answer 29

to | the square root of the sample size.

Question 30

If you find that the CI contains some unacceptable uncertainty, then you need to

Answer 30

run a new study with greater | sample size to narrow the CI .

Question 31

The power of a | statistical test is

Answer 31

``` the probability that it correctly rejects the null hypothesis when the null is false. ```

Question 32

`As power increases, the probability of | Type II errors

Answer 32

decreases

Question 33

Given the 1) sample size, 2) standard deviation, and | 3) size of effect, power is

Answer 33

the fraction of experiments | in which you would expect to find a significant result.

Question 34

• In practice, power | is the

Answer 34

``` fraction of experiments in which you would expect to find a statistically significant result. ```

Question 35

Which means that power can never be lower than α.

Answer 35

As effect size decreases to zero, power decreases to α. When effect size = 0, then null is true, and the only way to get a significant result is by chance (sampling error).