Cluster-robust inference and few clusters
What question does this help answer?
It lets you keep an ordinary regression as the model for the mean while making the uncertainty honest about the grouping. That is attractive when the clustering is a nuisance rather than the subject.
The problem it has, which is the point of this page
The ordinary sandwich estimator is justified asymptotically in the NUMBER OF CLUSTERS, not the number of people. With few clusters it underestimates standard errors, producing intervals that are too narrow and p-values that are too small, and adding more participants to the existing clusters does not fix it. The correction is least reliable exactly where it is most needed.
What to do instead when clusters are few
Use a small-sample correction rather than the plain sandwich. The CR2 bias-reduced linearization estimator, paired with Satterthwaite degrees of freedom estimated from the data, performs acceptably with as few as ten clusters. Kenward-Roger serves the same purpose for mixed models. A third option is to abandon individual-level inference and analyse CLUSTER-LEVEL SUMMARIES, which is simple, conservative and easy to explain. Randomization inference is a fourth, where the design was genuinely randomized.
Why Mallard warns about it
Mallard warns when a clustered design uses ordinary sandwich or GEE inference with fewer than about 30 clusters, and blocks below 15, unless a small-sample approach is named. The threshold is not magic; it marks the region where the approximation stops holding.
Key assumptions and design requirements
- The number of clusters is stated, because it determines whether the correction is trustworthy.
- Below about 30 clusters, a small-sample correction is NAMED: CR2, Satterthwaite, Kenward-Roger, or cluster-level summaries.
- Degrees of freedom are addressed, not left at the default; coverage depends on them as much as on the variance.
- Clustering is defined at the level at which the exposure or the design induces it, not at the most convenient level.
- The choice between cluster-robust inference and a random-effects model is stated, since they target different estimands under a nonlinear link.
What to watch out for
The ordinary sandwich estimator is asymptotic in the NUMBER OF CLUSTERS, not the number of people, so it underestimates standard errors exactly where clusters are few and the correction is most needed. Adding more participants to existing clusters does not help. A cluster-robust variance with fifteen clusters and no small-sample correction is a confident number with nothing behind it.
Clinical example
Question: Does a checklist reduce complications, in a study of twelve surgical units? Analysis: A cluster-robust variance with twelve clusters and no correction would be optimistic. Use the CR2 estimator with Satterthwaite degrees of freedom, or analyse the twelve unit-level rates directly, and say which was done.
- Using cluster-robust standard errors when analyzing group-randomized trials with few clusters (opens in a new tab) · Behav Res Methods · doi 10.3758/s13428-021-01627-0
- Determinants of the intracluster correlation coefficient in cluster randomized trials: the case of implementation research (opens in a new tab) · Clin Trials · doi 10.1191/1740774505cn071oa