Ask Mallard
Analysis method

Holm adjustment

5 min read

Holm adjustment tests a defined family from smallest to largest p-value, controlling the chance of even one false rejection.

Fit

When to use it, and when not to

Use it when

  • You need family-wise error control for a prespecified family of confirmatory tests.
  • Tests may be dependent, but each individual p-value is valid.

Not when

Before you start

What you need

Decide these first
  1. The full family of hypothesesDefine which claims belong together before seeing results. Include all tests in that family, not just favorable ones.
  2. Valid p-values and family-wise error levelRetain test labels and specify the error level before applying the adjustment.
It rests on these
  1. Each true null has a valid p-value.How to check: Check each model, sampling design and test. Multiplicity adjustment cannot repair bias or ignored clustering.
  2. The family includes every test supporting the joint claim.How to check: Check the protocol for outcomes, subgroups, time points and data-dependent searches.
  3. Holm permits arbitrary dependence between valid tests.How to check: Independence is not required. Repeated looks still need valid sequential p-values or a planned monitoring method.

Plain language

How it works

Family-wise error is the chance of rejecting at least one true null in the family. Sort m p-values from smallest to largest. At rank i, compare with the chosen error level divided by m−i+1. Reject and continue only while comparisons pass. Stop at the first failure.

Adjusted p-values

Multiply each sorted p-value by m−i+1. Take the running maximum and cap at one. Restore original labels. Holm needs valid p-values, but not independent tests. A family-wise level of 0.05 limits the chance of any false rejection to five in 100 repeated studies.

Step down and stop
Holm step-down: stop at the first failure. Repeat the comparison after each rejection. Valid individual p-values permit arbitrary dependence. A failure stops all remaining rejections.Holm step-down: stop at the first failureFull family and chosen error levelSort p-values from smallest to largestCompare with level divided by testsremainingReject and move to next rankStop rejecting all remaining hypothesesPassFailRepeat the comparison after each rejection. Valid individual p-values permit arbitrary dependence. Afailure stops all remaining rejections.
Conceptual diagram. It shows the structure of the idea, not data from any study.

Worked example

A worked example

Invented for illustration
Setting
An invented study defines five confirmatory outcomes as one family.
Question
Which null hypotheses can be rejected at family-wise error level 0.05?
Approach
Sorted p-values are 0.008, 0.012, 0.031, 0.032 and 0.200. Compare them in order with 0.05 divided by the number still untested.
Result
Holm-adjusted p-values are 0.040, 0.048, 0.093, 0.093 and 0.200. Reject the first two null hypotheses.
What it means
The third comparison fails: 0.031 exceeds 0.05/3. Stop there; later tests cannot restart rejection.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

Holm controls the family-wise error rate only for the full, scientifically defined family and valid individual p-values. Stop at the first failed comparison. It does not repair biased tests or permit repeated unplanned looks at accumulating data.

  • Continuing after the first failed comparison.Instead: Stop rejecting at that point. The ordered procedure is step-down.
  • Multiplying each p-value without enforcing monotonicity.Instead: Take the running maximum of the ordered products, cap at one, then restore original test labels.
  • Using ordinary confidence intervals as simultaneous intervals.Instead: Label unadjusted intervals honestly. Use a compatible simultaneous method when joint coverage is required.

Inside Mallard

How Mallard uses this

Mallard includes this method in Learn search and the Atlas. The method registry links its multiplicity-adjustment entry here. Mallard does not execute the adjustment.

Code example in Mallard: Multiplicity adjustment: Holm, Hochberg and Benjamini-Hochberg (multiplicity-adjustment).
Atlas node analysis.holm, the id a plan cue records.

How to report it

Report the prespecified family, every test, unadjusted and Holm-adjusted p-values, family-wise error level and decisions. Report effects with clearly labeled intervals; adjusted p-values alone do not give simultaneous interval coverage. A reporting guideline is not a design manual: use it to make the study transparent, not as a substitute for careful design. Check the target journal's own requirements as well, which this page cannot know.

Code for this method
Loading…

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. A simple sequentially rejective multiple test procedure (opens in a new tab) · Scandinavian Journal of Statistics 6:65–70
  2. A graphical approach to sequentially rejective multiple test procedures. (opens in a new tab) · Stat Med · doi 10.1002/sim.3495
  3. A less conservative method to adjust for familywise error rate in neuropsychological research: the Holm's sequential Bonferroni procedure. (opens in a new tab) · NeuroRehabilitation · doi 10.3233/NRE-130893
Prepared with AI assistance. PubMed records checked on 2026-11-07 for title, authors, year and DOI: record verified, confirm it supports this use. Holm supports the ordered procedure and family-wise guarantee. Bretz and colleagues develop sequentially rejective Bonferroni procedures; Eichstaedt and colleagues illustrate Holm arithmetic. Holm (1979) is not in PubMed. No independent professional or methodological approval is claimed.
Last source review: 2026-11-07
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.