Bonferroni correction
Bonferroni tests each hypothesis at the family error level divided by the number of tests, or multiplies each p-value by that number.
Fit
When to use it, and when not to
Use it when
- You want to limit the chance of any false rejection across a defined family.
- You need a simple single-step procedure valid with dependent tests.
Not when
- One planned test supports one claim with no selection. Use its ordinary significance level. Read: Confidence intervals and p-values
- You can use a step-down procedure for family-wise control. Consider Holm adjustment, which is never less powerful. Read: Multiple testing and multiplicity
- Your aim is discovery across many candidates. Consider False discovery rate control and state that different error goal. Read: Multiple testing and multiplicity
Before you start
What you need
- The full family of hypothesesCount all tests supporting the shared claim, including planned tests with unremarkable results.
- The family error levelChoose the tolerated probability of any false rejection before looking at the results.
- Valid unadjusted p-valuesEach underlying test must suit the design, estimator and assumptions.
- Each true-null p-value is valid at its allocated level.How to check: Check the underlying tests. Multiplying an invalid p-value cannot make its inference valid.
- The specified family captures the selection behind the claims.How to check: Account for repeated looks, outcomes and analysis searches. Tests need not be independent.
Plain language
How it works
Let m be the planned number of tests and alpha the family error level. Reject a hypothesis when its raw p-value is at most alpha divided by m. Equivalently, compare min(1, m times p) with alpha. Bonferroni bounds the chance of any false rejection by the sum of true-null error probabilities. No independence assumption is needed.
For simultaneous intervals, allocate alpha divided by m to each interval error probability. Coverage is at least one minus alpha when each component interval meets its guarantee. Holm adjustment can reject more while keeping family-wise control. False discovery rate control targets the expected fraction of false rejections, a different goal. Choose that goal in the multiplicity overview.
Worked example
A worked example
- Setting
- A trial prespecifies a family of 4 benefit hypotheses and a family error level of 0.05.
- Question
- Which hypotheses pass Bonferroni control?
- Approach
- The unadjusted p-values are 0.003, 0.012, 0.020 and 0.400. Multiply each by the family size and cap at one.
- Result
- The test threshold is 0.0125. Adjusted p-values are 0.012, 0.048, 0.080 and 1.000. The first 2 hypotheses reject at 0.05.
- What it means
- Two-sided marginal 98.75% intervals would give at least 95% simultaneous coverage if each interval has its stated coverage. Independence is not required.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
Define the test family before seeing results. Bonferroni controls family-wise error with valid p-values under any dependence, but can lose power. It cannot repair invalid tests or selective reporting.
- Counting only tests with small p-values.Instead: Keep the prespecified family size. Dropping unfavorable tests defeats the error guarantee.
- Comparing adjusted p-values with the divided threshold.Instead: Compare adjusted p-values with the family error level, or raw p-values with the divided threshold. Do not adjust twice.
- Assuming a non-rejection proves no effect.Instead: Report effect estimates and intervals. The correction can reduce power, especially with many related tests.
Inside Mallard
How Mallard uses this
Plans do not link to this method yet.
analysis.bonferroni, the id a plan cue records.How to report it
Report the prespecified family, number of tests, family error level, raw and adjusted p-values, and the correction. Label ordinary and simultaneous intervals clearly. A reporting guideline is not a design manual: use it to make the study transparent, not as a substitute for careful design. Check the target journal's own requirements as well, which this page cannot know.
Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Adjusting for multiple testing when reporting research results: the Bonferroni vs Holm methods (opens in a new tab) · Am J Public Health · doi 10.2105/ajph.86.5.726
- Some comments on frequently used multiple endpoint adjustment methods in clinical trials (opens in a new tab) · Stat Med · doi 10.1002/(sici)1097-0258(19971130)16:22<2529::aid-sim692>3.0.co;2-j
- Multiple significance tests: the Bonferroni method (opens in a new tab) · BMJ · doi 10.1136/bmj.310.6973.170