Chapters 22-23

Question 1

The problem of multiple comparisons

Answer 1

the most difficult challenge of statistics

-it is impossible to interpret P values without knowing how many comparisons

Question 2

Assuming two comparisons, what is the chance of obtaining at least one statistically significant conclusion (false positives) by chance?

Answer 2

1-0.9025=0.0975

Question 3

Assuming two comparisons, what is the chance of obtaining zero statistically significant conclusion (false positives) by chance?

Answer 3

0.95x0.95=0.9025

Question 4

With K independent
comparisons,
• The chance that
none will be
significant

Answer 4

is 0.95^K

Question 5

The chance that one or
more will be significant
when null is actually
true =

Answer 5

1- 0.95^K

Question 6

Multiple
comparisons can
be used to
generate
hypotheses,
• But not

Answer 6

in
experiments to
test hypotheses.

Question 7

Simply report uncorrected P values but clearly state

Answer 7

how many comparisons were made and that no

corrections were made to P values.

Question 8

Or designate one test as the

Answer 8

primary outcome and
all others (secondary outcomes) as exploratory
analyses.
• must be a priori.

Question 9

Or if all of the comparisons produced similar

significant results,

Answer 9

Then all would lead to the same conclusion
whether just one or many comparisons were
performed.

Question 10

Familywise Error Rate

Answer 10

Previously, α was
used to represent
the Type I Error rate
per comparison.
• With multiple
comparisons, we
need a familywise
error rate.

Question 11

Here the significance level is redefined to be the

Answer 11

chance of obtaining one or more statistically

significant conclusions if all null hypotheses are true.

Question 12

Bonferroni Correction

Answer 12

The most common approach is
to divide the value of α by the
number of comparisons.
• For 20 comparisons:
• 0.05/20 = 0.0025
• i.e. α /K (where K is the
number of comparisons)

Question 13

Now there is only a 5% chance
of seeing (with a Bonferroni correction)

Answer 13

one or more
significant results (if all nulls
are true) in the whole study.

Question 14

Defining Groups With the Data

Answer 14

Groups must be
defined before
seeing the data.
• If defined by the
data, then multiple
comparisons have
already occurred.

Question 15

When something unusual
happens, it is tempting to
calculate the probability
that this event could
happen in this place.
• But what you really need to
know is

Answer 15

the probability that
this event could happen in
one of the places that you
can observe.

Question 16

Researchers may
develop a model to
explain an event
using

Answer 16

multiple
explanatory
(independent)
variables).

Question 17

If you choose individual explanatory variables from a large set of
candidates, and then just use the best in the model, you have
performed multiple comparisons.

Answer 17

Some of those variables were

probably associated with the dependent variable just be chance.

Question 18

If you keep adding
samples until you get
a significant result,
but then stop adding
samples every time
you get significant
results.

Answer 18

You have bias the results in favor if Type I Errors.

Question 19

Data Torture/Data Diving

Answer 19

Example:
• Just keep running
different analyses
until you get a
significant result.
• This is multiple
comparisons.
The chances of a Type I Error are substantially
increased.