Choosing variables for a model
Variable selection decides which factors enter a model, and the right way to choose depends on what the model is for: an effect, a prediction or a description.
Fit
When it matters, and when it does not
It matters when
- You are fitting a regression with more than one factor, and must decide which factors go in.
- You want the effect of an exposure, and must choose the confounders to adjust for.
- You are building a prediction model from many candidate predictors, and need one that holds up in new patients.
It matters less when
- You want the effect of an exposure and plan to let an algorithm pick the confounders. Choose them from a causal diagram instead. Read: Directed acyclic graphs
- The goal is prediction with many candidate predictors. Choose a shrinkage method and validate the full procedure. Read: Choosing a penalty
- The protocol already fixes the model. Report that model, whatever its coefficients turn out to be. Read: Prespecification and the analysis plan
Before you start
What to check in your study
- What the model is fortargetAn effect, a prediction or a description. Each needs a different way of choosing factors.
- For an effect, a causal diagram drawn before the analysisIt shows which factors are confounders to adjust for, and which are mediators or colliders to leave out.
- For prediction, the candidate predictors, counted as parametersChoose them from knowledge and from what will be known when the prediction is made. A category with several levels counts as several.
- How much information the data holdoutcomeTypeFor a yes-or-no or time-to-event outcome, count events, not patients. Events per variable is only a floor; Riley's criteria size a prediction model.
- For an effect, the causal diagram reflects what is known about how the factors relate.How to check: Draw it with clinical colleagues before seeing the data. Name each factor's role: confounder, mediator or collider.
- Confounders are measured before the exposure.How to check: A factor measured after the exposure may be one of its results. Leave it out of the adjustment set.
- For prediction, the chosen set would be chosen again in a similar sample.How to check: Repeat the whole selection in bootstrap samples, and report how often each predictor is kept.
Plain language
How it works
For an effect, the adjustment set comes from knowledge of the causal structure, usually drawn as a causal diagram. For prediction, any factor that improves accuracy in new patients may enter. For a description, neither rule applies. Using a prediction-style algorithm to pick confounders lets in mediators and colliders, which adds the very bias adjustment was meant to remove.
Prespecify a causal adjustment set from clinical knowledge. For prediction, compare shrinkage methods and assess the whole procedure in resampling. Ridge retains the candidates; lasso can select a subset. Neither turns selected predictors into causes. The single-method pages explain fitting and interpretation.
Worked example
A worked example
- Setting
- Records of 3,000 adults discharged from medical wards, with 25 candidate predictors of readmission.
- Question
- Which predictors should a readmission risk score use?
- Approach
- Backward elimination by P value is run on all the data. The same selection is then repeated in each of 200 bootstrap samples.
- Result
- On all the data, it kept 9 predictors, including recent falls. Recent falls was kept in only 70 of the 200 bootstrap samples (35 in 100).
- What it means
- A predictor dropped in most resamples may not be chosen again in a new sample. Ridge retains candidates; lasso can still select an unstable subset.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
Stepwise selection is the default in most software and it is unreliable: the coefficients it keeps are biased away from zero, the p-values and confidence intervals it reports are wrong because they ignore that the model was chosen from the same data, and the set it selects is unstable, so a different sample of the same population yields a different model. The deeper error is treating this as one problem when it is two: selecting confounders for a causal question and selecting predictors for a prediction model follow different rules.
- Choosing confounders by whether each factor was significant on its own against the outcome.Instead: A factor's own P value says nothing about whether it is a confounder. Use a causal diagram.
- Adjusting for a factor because it moved the exposure's estimate by 10%.Instead: Mediators and colliders move the estimate too. The change-in-estimate rule cannot tell them from confounders.
- Dropping the exposure from the model because of its P value.Instead: The exposure stays in whatever its P value. It is the question being asked.
- Reporting a prediction model's performance on the data used to select its predictors.Instead: Repeat the selection in every bootstrap sample or cross-validation fold, and correct the performance for optimism.
Inside Mallard
How Mallard uses this
Mallard's causal diagram lists the confounders it proposes to adjust for, labeled as a proposed set for you to confirm. A plan that lists one factor as both confounder and mediator is flagged as blocking. Plans do not link to it yet.
concept.choosing_variables, the id a plan cue records.Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Variable selection - A review and recommendations for the practicing statistician (opens in a new tab) · Biom J · doi 10.1002/bimj.201700067
- Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes (opens in a new tab) · Stat Med · doi 10.1002/sim.7992
- Principles of confounder selection (opens in a new tab) · Eur J Epidemiol · doi 10.1007/s10654-019-00494-6