Multiple Testing

Multiple testing is the consideration of several statistical hypothesis tests within a common inferential framework. In statistics, testing each hypothesis separately at a conventional threshold does not generally preserve that same error rate for the collection. Multiple-testing procedures define an appropriate collective error criterion and adjust rejection thresholds, p-values, or simultaneous intervals accordingly. Multiple comparisons between group means are a prominent special case, but the framework also covers collections of outcomes, parameters, and scientific claims. (itl.nist.gov)

Why multiplicity matters

A Type I error occurs when a true null hypothesis is rejected. A test conducted at significance level $\alpha$ limits the probability of this error individually, assuming its statistical requirements are satisfied. Repeating tests creates additional opportunities for false rejection; the individual level does not automatically become the overall level. (fda.gov)

For $m$ true null hypotheses whose rejection events have statistical independence, each with exact error probability $\alpha$, the probability of at least one false rejection is

For example, with $m=20$ and $\alpha=0.05$, this is approximately $0.642$. Independence is essential to this calculation: dependent tests need not follow the formula. The expected number of false rejections is nevertheless $m\alpha$ when each test has exact size $\alpha$, regardless of dependence. These are distinct measures of error, rather than competing calculations of the same quantity. (math.tau.ac.il)

Families and error criteria

A family is the collection of hypotheses over which collective error control is sought. Its boundaries depend on the scientific claims being evaluated, not merely on which tests happen to share a dataset. Multiple endpoints, treatment comparisons, time points, and subgroup analyses can create multiplicity within a study. Grouping and ordering hypotheses are therefore components of experimental design and analysis planning. (fda.gov)

Let $V$ denote the number of rejected true null hypotheses and $R$ the total number of rejections. Two central criteria are:

• Family-wise error rate (FWER):

  $$\operatorname{FWER}=\Pr(V\geq1).$$

• False discovery rate (FDR):

  $$\operatorname{FDR} =\mathbb{E}\!\left[\frac{V}{\max(R,1)}\right].$$

FWER concerns whether any false rejection occurs; FDR concerns the expected proportion of false rejections among all rejections, taking that proportion as zero when none occur. Strong FWER control applies under every configuration of true and false hypotheses. Weak control applies only when all null hypotheses are true. (stat.purdue.edu)

FDR never exceeds FWER, and the two coincide when all null hypotheses are true. FDR control at $0.05$ does not guarantee that at most 5% of the rejections in any particular dataset are false, nor does it assign each rejected hypothesis a 5% probability of being false. It is an expectation over repeated sampling. (stat.purdue.edu)

Family-wise error procedures

The Bonferroni correction rejects hypothesis $i$ when $p_i\leq\alpha/m$. Equivalently, its adjusted p-value is $\min(1,mp_i)$. Its justification uses the union bound, so independence is unnecessary. Provided each individual p-value is valid under its null hypothesis, the procedure strongly controls FWER. (stat.ethz.ch)

The Holm procedure orders the p-values as $p_{(1)}\leq\cdots\leq p_{(m)}$. Starting with the smallest, it compares $p_{(i)}$ with $\alpha/(m-i+1)$, rejecting until the first comparison fails. This step-down procedure also strongly controls FWER under arbitrary dependence and rejects every hypothesis that ordinary Bonferroni would reject, potentially more. Less stringent thresholds can improve statistical power, the probability of detecting a false null hypothesis. (stat.ethz.ch)

Specialized procedures exploit the structure of particular comparison families. Following analysis of variance, Tukey’s procedure addresses all pairwise mean comparisons, while Scheffé’s method addresses all linear contrasts. An omnibus rejection establishes that the means are not all equal; it does not identify which individual differences are responsible. These procedures require their respective model assumptions. (itl.nist.gov)

False-discovery-rate procedures

The Benjamini–Hochberg procedure, introduced in 1995, orders the p-values and finds the largest index $k$ satisfying

where $q$ is the target FDR level. It rejects hypotheses corresponding to $p_{(1)},\ldots,p_{(k)}$; if no index qualifies, it rejects none. Unlike a step-down procedure, it does not stop at the first failed comparison. Its original proof established FDR control for independent test statistics. (rss.onlinelibrary.wiley.com)

The same procedure also controls FDR under a specified positive-dependence condition, called positive regression dependence on a subset. Merely observing positive correlation is not a general substitute for verifying that condition. The Benjamini–Yekutieli procedure accommodates arbitrary dependence by replacing $q$ with $q/H_m$, where $H_m=\sum_{j=1}^{m}1/j$. This broader guarantee comes with more restrictive thresholds. (math.tau.ac.il)

Simultaneous inference and interpretation

Multiplicity also affects confidence intervals. Separate 95% intervals do not generally provide 95% probability of covering every corresponding parameter simultaneously. Bonferroni intervals with individual coverage at least $1-\alpha/m$ provide simultaneous coverage of at least $1-\alpha$. This joint guarantee is different from the coverage of any one interval. (itl.nist.gov)

The interpretation of an adjusted result depends on the specified family, procedure, error criterion, and assumptions. FWER and FDR procedures answer different questions and need not produce the same rejection set. Adjustment controls a defined error rate; it does not establish the magnitude or scientific importance of an effect. When comparisons are selected after examining the data, the selection process also matters: applying a correction only to the selected comparisons does not generally reproduce the guarantees of a procedure designed for the complete search. (stat.ethz.ch)