Penalized regression and shrinkage
What question does this help answer?
How to build a model that predicts well in NEW patients, when the number of candidate predictors is large relative to the amount of outcome information.
The problem it addresses
An ordinary model fitted to limited data is too confident: it produces predictions that are too extreme, high risks too high and low risks too low. That is overfitting, and it shows up as a calibration slope below one in new data. Shrinking the coefficients pulls the predictions back toward the average and typically improves accuracy where it matters.
The main methods
- Ridge: shrinks all coefficients smoothly, keeps them all. Good when many predictors each contribute a little.
- Lasso: shrinks and can set coefficients exactly to zero, so it selects as well as shrinks. Attractive and unstable when predictors are correlated.
- Elastic net: a mixture of the two.
- Uniform shrinkage: estimate one factor and apply it to every coefficient after fitting, which is simple and transparent.
- Firth's correction: addresses separation and small-sample bias in logistic regression, with a small shrinkage effect and unusually low variability.
Choosing the tuning parameter
The strength of the penalty is usually chosen by cross-validation. That choice is itself an estimate with uncertainty, and any performance figure computed on the same data used to tune is optimistic. Nesting the tuning inside the resampling is the correct construction, and it is frequently skipped.
Why Mallard may recommend it
Mallard may suggest penalization when a prediction model is proposed with many candidate predictors and limited outcome events, alongside a sample size calculation aimed at limiting overfitting in the first place.
What to watch out for
Penalization is not a licence to fit a large model to a small dataset. The amount of shrinkage is itself estimated from the data, and in exactly the situation where shrinkage is most needed, a small sample, that estimate is most uncertain, so the resulting model can be badly miscalibrated in either direction. It also does not fix a poorly specified model or make a causal coefficient interpretable: a shrunk coefficient is deliberately biased, which is fine for prediction and not for effect estimation.
Clinical example
Question: Build a risk score for readmission from thirty routinely collected variables. Analysis: Ridge or elastic-net logistic regression with the penalty tuned inside a cross-validation loop, reported with a calibration slope from data not used in fitting, and sized so that shrinkage is a refinement rather than a rescue.
- Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small (opens in a new tab) · J Clin Epidemiol · doi 10.1016/j.jclinepi.2020.12.005
- Regression shrinkage methods for clinical prediction models do not guarantee improved performance: Simulation study (opens in a new tab) · Stat Methods Med Res · doi 10.1177/0962280220921415