When performing multiple statistical tests of significance, we can expect to produce some false positives. Below we discuss a few common ways of adjusting the p values of our individual tests to achieve some criterion over the family of tests.
One type of correction aims to reduce the family-wise error rate (FWER), the probability of at least one false positive.
The p-value gives us the probability of a false positive under the null-hypothesis of a single statistical test. If we perform $N$ independent such tests, the probability of at least one false positive is $$ p_N = 1 – (1 – p)^N.$$
Suppose we want this value to be less than some desired value $\alpha$. What $p$ value should we use in our tests?
$$ 1 – (1 – p)^N = \alpha \implies (1 – \alpha)^{1/N} = 1 – p \implies p = 1 – (1 – \alpha)^{1/N}.$$ This is called the Sidak correction.
A slightly simpler correction is the Bonferroni correction. One way to arrive at it is to start with our expression relating $p_N$ and $\alpha$ above, and then take logarithms to arrive at $$ (1 – \alpha) = (1 – p)^N \implies \log (1 – \alpha) = N \log (1 – p).$$ Since $\alpha$ and $p$ are small, we next use the approximation $ \log (1 – \veps) \approx -\veps$ to get $$ \alpha = N p \implies p = {\alpha \over N}.$$
We assumed our tests were independent, but the Bonferroni correction is much more general. To see this, consider running two tests. Let $A$ be the event of a false alarm in the first, and $B$ the same in the second. The probability of at least one false alarm is $$ 1 – P(\neg A \cap \neg B)= P(A \cup B) \le P(A) + P(B),$$ by the union bound.
So for $N$ tests, each with a false alarm probability of $p$, the probability of at least one false alarm in the full set of tests is less than or equal to $ N p$. Therefore, if we want this probability to be less than $\alpha$, we need $p < \alpha / N$, the Bonferroni correction.
Both of these corrections reduce the probability of false positives by making the threshold for detection more strict. This can be overly conservative and hurt our chances of finding true positives.
Rather than controlling false positives alone, the Benjamini-Hochberg procedure controls the false discovery rate, which is defined as $$ FDR = {FP \over FP + TP}.$$ Because the FDR takes true positives into account, it’s not necessarily reduced by reducing the number of false positives if that also reduces the true positives too much.
The correction procedure is simple. We have $m$ comparisons, each with their own p-values. We rank them by p-value from lowest to highest. Now consider the $k$’th such ranked value, $p_k$. If we accepted the corresponding $k$ tests as discoveries, our expected number of false discoveries across our $m$ comparisons would be $p_k m$. Our total claimed number of discoveries is $k$, since $k$ of the tests, by the ranking, would be at or below the threshold. If our desired FDR is $ \le q$, we would then need $$ FDR = {m p_k \over k} \le q \implies p_k \le {q k \over m}.$$ So we find the largest value for $k$ for which this holds and use the corresponding $p_k$ as our data-dependent threshold, and reject all tests with $p \le p_k$.
$$\blacksquare$$
Leave a Reply