10. Large-scale hypothesis testing

Question 1

What problem does Hypothesis testing adress?

Answer 1

Hypothesis testing addresses decision problems (usually about comparisons)
in statistical inference.

Question 2

What is the standard procedure?

Answer 2

1. Vague idea
2. Precise hypotheses
3. Gather data
4. Perform a statistical test to reject or fail to reject the null hypothesis

Question 3

Can you draw a distribution of test statistic under null hypothesis

Question 4

Explain the two types of errors we can make

Answer 4

Type II error
Null hypothesis should
have been rejected,
but wasn’t.

Type I error
Null hypothesis was
rejected, but shouldn’t
have been.

Question 5

What can be the problem of running too many tests?

Answer 5

Running thousands of hypothesis tests will give
hundreds of false positives (Type I errors).

Question 6

What can we then do?

Answer 6

▶ Very common workaround for multiple testing:
Divide significance level α by number of tests N,
test each hypothesis at level α/N.

Under the null hypthesis the p-values have a uniform disutrbution

Question 7

What is holm’s procedure?

Answer 7

Procedure to control FWER at level α.
Uniformly more powerful than Bonferroni bound.
Procedure:
1. Order the observed p-values from smallest to largest:
Let i0 be the smallest i such that p p(i) > α/(N − i + 1).
3. Reject all null hypotheses for i < i0 and accept all for i ≥ i0.

Question 8

What is the benjamini-Hochberg procedure

Answer 8

1. Order the observed p-values from smallest to largest:
2. Let imax be the largest index for which
3. Reject all null hypotheses for i ≤ imax, accept the rest.

Question 9

What the difference between FDR and Fdr

Answer 9

Now compare:
(Frequentist) FDR:

Empirical Bayes) Fdr (hat):

Question 10

1. Is controlling a rate (FDR) as meaningful as controlling a probability?

Answer 10

FDR control does control the Bayes posterior probability
of the null hypothesis being true.