Subgroup analyses
What question does this help answer?
Whether the average result applies to the patient in front of you. It is a genuinely important question and it is also the place where trials most reliably produce findings that do not replicate.
Test the interaction, not each half
Significant in men, not significant in women is not evidence that the effect differs. Two estimates can sit on opposite sides of a significance threshold while being statistically indistinguishable from each other, which happens constantly because subgroups have roughly half the sample and therefore much less power. The question requires a formal test of the difference between the subgroup effects.
What a subgroup forest plot looks like
What makes a subgroup finding credible
- It was prespecified, with the direction of the expected difference stated, before any results were seen.
- There are few subgroups, and the one in question was among the small number nominated in advance.
- The interaction test itself is significant, ideally strongly.
- The subgroup variable was measured at baseline, before randomization.
- There is a mechanism that predicts the difference, and consistency with other studies.
- The effect is in the same direction in both subgroups, differing in size. A qualitative interaction, where the treatment helps one group and harms another, is a much stronger claim needing much stronger evidence.
Power is the underlying problem
A trial sized to detect a main effect has very little power to detect an interaction, typically needing about four times the sample size for an interaction of the same magnitude. So a non-significant interaction test in a normal-sized trial says almost nothing, and a significant one is disproportionately likely to be a false positive or a large overestimate.
Why Mallard may recommend it
Mallard may include a prespecified subgroup analysis when a clinically motivated effect modifier exists, and will describe it as exploratory unless the study was sized for the interaction.
What to watch out for
A trial with ten subgroups has ten chances to find a difference that is not there, and the arithmetic is unforgiving: with ten independent subgroups, a false positive somewhere is more likely than not. The right test is the INTERACTION test, which asks whether the effects differ; comparing whether one subgroup reached significance and the other did not is a different and invalid comparison, and it is the one almost always made. Subgroups defined by anything measured AFTER randomization, response included, are not randomized comparisons at all.
Clinical example
Question: Does the treatment work better in patients with severe disease? Analysis: A prespecified interaction between treatment and baseline severity, reported as the interaction estimate with its confidence interval, and labelled exploratory since the trial was sized for the overall effect.
- Statistics in medicine--reporting of subgroup analyses in clinical trials (opens in a new tab) · N Engl J Med · doi 10.1056/NEJMsr077003
- Credibility of subgroup analyses by socioeconomic status in public health intervention evaluations: An underappreciated problem? (opens in a new tab) · SSM Popul Health · doi 10.1016/j.ssmph.2018.09.010