Chapters 18-20

Question 1

What does it mean if the results are statistically significant?

Answer 1

Means that the results would be surprising, but not impossible if the null hypothesis were true. But we really care about effect size

Question 2

The relationship between the P value and size of the effect depends on…

Answer 2

sample size

Question 3

Small effect can be significant with…

Answer 3

a large sample.

Question 4

Relative risk

Answer 4

The ratio of the treated group to control group (larger means greater effect size)

Question 5

False Discovery Rate (FDR)

Answer 5

Given that the result is significant, the probability that the null hypothesis is actually true. This cannot be truly known.

Question 6

The FDR probability cannot usually be determined because…

Answer 6

every probability calculation must be based on a model that can be expressed mathematically.

Question 7

FDR=

Answer 7

Type I Errors/Total Decisions to Reject Null

Question 8

However, our interpretation always involved at least an informal process of trying to estimate the FDR.

Answer 8

Examples 1, 2, and 3

Question 9

Example 1 of informally estimating the FDR:

Answer 9

P value is

Question 10

Example 2 of informally estimating the FDR:

Answer 10

P value is

Question 11

Example 3 of informally estimating the FDR:

Answer 11

P value is P value is

Question 12

What if you somehow know the probability that the null hypothesis is false?

Answer 12

Such knowledge would be an example of a prior probability

Question 13

Perhaps a prior probability could be based on…

Answer 13

your previous experiences (e.g. 50% of people brought to trial int he past really were criminals)

Question 14

Power

Answer 14

The power of a statistical test is the probability that it correctly rejects the null hypothesis when the null is false

Question 15

Power can be calculated if you know:

Answer 15

1) sample size 2) standard deviation 3) size of effect of interest

Question 16

Power=

Answer 16

Decisions to reject null/# times null is false

Question 17

FDR=

Answer 17

type I errors/total # decisions to reject null

Question 18

The Bayesian approach to FDR

Answer 18

Can have a big impact on interpretation!
• Example 1: You have
a drug developer
who has a track
record of 80%.
• Example 2: You have
experience with
traditional herbal
remedies producing
desired effect level
10% of the time.

Question 19

The Bayesian Approach to FDR (2)

Answer 19

Example 1:
• FDR =
• Example 2:
• FDR =
Effect of Prior Probability (0.10) on Interpretation
• In drug trials with Effect of Prior Probability (0.80) on Interpretation
the same level of
statistical power,
• Example 1:
• FDR = 10/650 = 0.15
• Example 2:
• FDR = 45/125 = 0.36

Question 20

FDR:

Answer 20

proportion of significant results that are really type I errors

Question 21

The Bayesian Approach to FDR is…

Answer 21

no more accurate than your prior probabilities

Question 22

Not statistically significant means…

Answer 22

only that the P value is larger than the present threshold. Does not prove that the null hypothesis is true.

Question 23

Keep in mind, the relationship between the P value and

size of the effect depends on sample size

Answer 23

Large effect can be nonsignificant with a small sample.

Question 24

CI helps put the data in perspective

Answer 24

- Testing whether the
amount of epinephrine
receptors on cells is
different in people with
high blood pressure.
- Used platelets as a
convenient ‘cell’ type
to measure.
- Means were similar.
- P value = 0.81
If you stop at this point and publish only these
results, you might imply that the null is true.

Question 25

Example 1:

• Means were similar.
• P value = 0.81

Answer 25

- The observed means
only differ by +6.
 - Out of total mean of 260.
- But 95% CI for the
difference is -45 to +57.
- Approximately ±20%.
- Would it matter if the
true difference is
actually 20%?
-You need to decide whether this level of uncertainty matters in the scientific/medical context

Question 26

Example 2:

Answer 26

A large study tests the
impacts of sonograms.
-They are expensive
but common.
Two treatments:
1) 2 routine sonograms
2) 0 unless physician
saw a special need
Observations:
-count adverse health
outcomes for fetus.
Results pretty similar (P=0.86) with slightly higher
adverse outcomes when sonograms used more often.

Question 27

Outcomes similar:
- P value = 0.86
- Null: relative risk = 1.0
- 95% CI for relative risk:
 - 0.88 to 1.17
- So it is possible that
sonograms reduce risk by
12% or raise risk by 17%.
- But we definitely can’t
rule out the null
hypothesis.

Answer 27

So are they useful? What if the null was true (no

impact on safety)? Might they still be ‘useful’?

Question 28

How to Get Narrower CIs

Answer 28

.
• Unless you can
change the SD
(not usually),
then sample
size must be
increased.
• Rule of thumb:
- Increase sample
size by factor of 4
to decrease CI by
factor of 2.

Question 29

Recall: the width of the CI is inversely proportional

Answer 29

to

the square root of the sample size.

Question 30

If you find that the CI contains some
unacceptable uncertainty, then you
need to

Answer 30

run a new study with greater

sample size to narrow the CI .

Question 31

The power of a

statistical test is

Answer 31

the probability
that it correctly
rejects the null
hypothesis when
the null is false.

Question 32

`As power increases, the probability of

Type II errors

Answer 32

decreases

Question 33

Given the 1) sample size, 2) standard deviation, and

3) size of effect, power is

Answer 33

the fraction of experiments

in which you would expect to find a significant result.

Question 34

• In practice, power

is the

Answer 34

fraction of
experiments in
which you would
expect to find a
statistically
significant result.

Question 35

Which means that power can never be lower than α.

Answer 35

As effect size decreases to zero, power decreases to α.
When effect size = 0, then null is true, and the only
way to get a significant result is by chance (sampling error).