Multiple testing and multiplicity
What question does this help answer?
“If we test many outcomes, subgroups, time points, or model variants, how should we control and interpret the risk of chance findings?”
Why it matters
A study that examines many hypotheses can produce apparently significant findings by chance. The appropriate response depends on which claims are confirmatory, which are secondary, and which are exploratory.
How the chance of a false positive grows
How Mallard may use this concept
Mallard may flag multiplicity when a plan includes multiple primary outcomes, many pairwise comparisons, numerous subgroup interactions, repeated testing over time, or a large biomarker set.
Good practice
- Define the primary estimand or outcome clearly.
- Prespecify confirmatory hypothesis families.
- Use an adjustment approach appropriate to the inferential goal when adjustment is needed.
- Distinguish family-wise error control from false-discovery-rate approaches.
- Label exploratory analyses as exploratory.
- Interpret effect estimates and confidence intervals, not only adjusted p-values.
Common adjustment methods
- Bonferroni: test each hypothesis at the overall significance level divided by the number of tests. It controls the family-wise error rate, the chance of any false positive, whatever the dependence between the tests, and is the most conservative.
- Holm: order the p-values from smallest to largest and compare each with a progressively less strict threshold, stopping at the first that fails. It controls the family-wise error rate under the same conditions as Bonferroni and always rejects at least as many hypotheses, so there is little reason to prefer plain Bonferroni.
- Hochberg: the step-up counterpart, starting from the largest p-value. It is more powerful than Holm but requires the tests to be independent or positively dependent.
- Benjamini-Hochberg: controls the false discovery rate, the expected proportion of rejected hypotheses that are false, rather than the chance of any false positive, under independent or positively dependent tests. It suits screening many secondary or exploratory outcomes, and it is not a substitute for family-wise control of confirmatory claims.
- Fixed-sequence testing: test prespecified hypotheses in a fixed order, each at the full level, and stop at the first that fails. It is common for a primary outcome followed by key secondary outcomes in a trial.
What to watch out for
Multiplicity is not solved by reporting only the favorable results. Conversely, indiscriminate correction across unrelated exploratory questions can be unnecessarily conservative. The family of inferences should be scientifically defined.
Clinical example
A trial with one primary outcome, five secondary outcomes, and ten exploratory subgroups should not treat all sixteen analyses as equivalent confirmatory claims.
- The fundamentals of multiplicity adjustment in biostatistics (opens in a new tab) · J Natl Cancer Inst Monogr · doi 10.1093/jncimonographs/lgae050
- When to Adjust for Multiple Testing: A Unifying Guiding Principle (opens in a new tab) · Biom J · doi 10.1002/bimj.70148
- False discovery rate control is a recommended alternative to Bonferroni-type adjustments in health studies (opens in a new tab) · J Clin Epidemiol · doi 10.1016/j.jclinepi.2014.03.012