Holm adjustment
Holm adjustment tests a defined family from smallest to largest p-value, controlling the chance of even one false rejection.
Fit
When to use it, and when not to
Use it when
- You need family-wise error control for a prespecified family of confirmatory tests.
- Tests may be dependent, but each individual p-value is valid.
Not when
- You accept control of the expected false-discovery fraction for exploratory screening. Consider Benjamini-Hochberg instead. Read: False discovery rate
- You repeatedly inspect accumulating trial data. Plan sequential boundaries rather than applying Holm to unadjusted interim p-values. Read: Interim analyses and stopping rules
Before you start
What you need
- The full family of hypothesesDefine which claims belong together before seeing results. Include all tests in that family, not just favorable ones.
- Valid p-values and family-wise error levelRetain test labels and specify the error level before applying the adjustment.
- Each true null has a valid p-value.How to check: Check each model, sampling design and test. Multiplicity adjustment cannot repair bias or ignored clustering.
- The family includes every test supporting the joint claim.How to check: Check the protocol for outcomes, subgroups, time points and data-dependent searches.
- Holm permits arbitrary dependence between valid tests.How to check: Independence is not required. Repeated looks still need valid sequential p-values or a planned monitoring method.
Plain language
How it works
Family-wise error is the chance of rejecting at least one true null in the family. Sort m p-values from smallest to largest. At rank i, compare with the chosen error level divided by m−i+1. Reject and continue only while comparisons pass. Stop at the first failure.
Multiply each sorted p-value by m−i+1. Take the running maximum and cap at one. Restore original labels. Holm needs valid p-values, but not independent tests. A family-wise level of 0.05 limits the chance of any false rejection to five in 100 repeated studies.
Worked example
A worked example
- Setting
- An invented study defines five confirmatory outcomes as one family.
- Question
- Which null hypotheses can be rejected at family-wise error level 0.05?
- Approach
- Sorted p-values are 0.008, 0.012, 0.031, 0.032 and 0.200. Compare them in order with 0.05 divided by the number still untested.
- Result
- Holm-adjusted p-values are 0.040, 0.048, 0.093, 0.093 and 0.200. Reject the first two null hypotheses.
- What it means
- The third comparison fails: 0.031 exceeds 0.05/3. Stop there; later tests cannot restart rejection.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
Holm controls the family-wise error rate only for the full, scientifically defined family and valid individual p-values. Stop at the first failed comparison. It does not repair biased tests or permit repeated unplanned looks at accumulating data.
- Continuing after the first failed comparison.Instead: Stop rejecting at that point. The ordered procedure is step-down.
- Multiplying each p-value without enforcing monotonicity.Instead: Take the running maximum of the ordered products, cap at one, then restore original test labels.
- Using ordinary confidence intervals as simultaneous intervals.Instead: Label unadjusted intervals honestly. Use a compatible simultaneous method when joint coverage is required.
Inside Mallard
How Mallard uses this
Mallard includes this method in Learn search and the Atlas. The method registry links its multiplicity-adjustment entry here. Mallard does not execute the adjustment.
multiplicity-adjustment).analysis.holm, the id a plan cue records.How to report it
Report the prespecified family, every test, unadjusted and Holm-adjusted p-values, family-wise error level and decisions. Report effects with clearly labeled intervals; adjusted p-values alone do not give simultaneous interval coverage. A reporting guideline is not a design manual: use it to make the study transparent, not as a substitute for careful design. Check the target journal's own requirements as well, which this page cannot know.
Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- A simple sequentially rejective multiple test procedure (opens in a new tab) · Scandinavian Journal of Statistics 6:65–70
- A graphical approach to sequentially rejective multiple test procedures. (opens in a new tab) · Stat Med · doi 10.1002/sim.3495
- A less conservative method to adjust for familywise error rate in neuropsychological research: the Holm's sequential Bonferroni procedure. (opens in a new tab) · NeuroRehabilitation · doi 10.3233/NRE-130893