Variable selection
Ask what the model is for, first
For a CAUSAL question the adjustment set follows from subject-matter knowledge about the causal structure, usually via a directed acyclic graph. For a PREDICTION model the aim is out-of-sample accuracy and any variable that helps is admissible. For a DESCRIPTIVE summary neither applies. Using a prediction-style algorithm to pick confounders is how mediators and colliders end up in causal models, introducing exactly the bias the adjustment was meant to remove.
Why data-driven selection misleads
Every selection step is a hypothesis test, and the inference that follows pretends none of them happened. The coefficients that survive are the ones that happened to look large in this sample, which is why they are systematically overstated, and the standard errors do not include the uncertainty about the model itself. The result looks more certain than it is, in a predictable direction.
Better approaches
- Prespecify the model from knowledge, and report it whatever the coefficients turn out to be.
- For confounding, build a causal diagram and adjust for a sufficient set; do not add or drop covariates on the basis of their p-values.
- For prediction with many candidates, penalized regression shrinks all coefficients rather than selecting a subset, which is more stable.
- Where selection is unavoidable, resample the whole selection process and report how often each variable is chosen; instability that large is worth showing.
- Never select on the significance of the exposure of interest, which is the most consequential form of this error.
How many variables can be afforded
The limit is set by the amount of information, which for a binary or time-to-event outcome means the number of EVENTS rather than the number of participants. Events-per-variable rules of thumb are a floor, not a target; the modern approach computes what the data can support from the expected model performance.
Why Mallard may recommend a particular set
Mallard proposes an adjustment set from the causal structure the description implies, and names it as a PROPOSED set the investigator must confirm rather than an answer.
What to watch out for
Stepwise selection is the default in most software and it is unreliable: the coefficients it keeps are biased away from zero, the p-values and confidence intervals it reports are wrong because they ignore that the model was chosen from the same data, and the set it selects is unstable, so a different sample of the same population yields a different model. The deeper error is treating this as one problem when it is two: selecting confounders for a causal question and selecting predictors for a prediction model follow different rules.
Clinical example
Question: Does a medication reduce readmission? Analysis: Adjust for the confounders a causal diagram identifies, prespecified in the protocol, rather than running stepwise selection over thirty candidate variables and reporting whatever remained.
- Variable selection - A review and recommendations for the practicing statistician (opens in a new tab) · Biom J · doi 10.1002/bimj.201700067
- Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes (opens in a new tab) · Stat Med · doi 10.1002/sim.7992