Choosing a penalty
Choose ridge, lasso or elastic net for a prediction goal; consider Firth logistic regression for small-sample bias or separation.
Fit
When it matters, and when it does not
It matters when
- You need to limit unstable coefficients when building a prediction model.
- Your logistic model has separation: some predictors perfectly distinguish outcomes and ordinary estimates can diverge.
It matters less when
- You need a causal adjustment set. Choose confounders from the causal question, not from penalized selection. Read: Confounding
- You need to assess how well predictions work elsewhere. Plan validation. Read: Prediction model performance and validation
Before you start
What to check in your study
- The problem to solveDistinguish predictive accuracy, a compact model and finite logistic estimates.
- Predictor scalesChoose scaling before applying a penalty. Handle scaling within each training sample.
- How tuning will be assessedKeep test data out of fitting, feature selection and tuning.
- Validation reflects how the model will be used.How to check: Choose resampling units that preserve clustering and time order when needed.
- A finite estimate still needs enough information.How to check: For Firth fits, inspect uncertainty and prefer appropriate penalized likelihood inference over a claim of certainty.
Plain language
How it works
Ridge penalizes squared coefficients and usually keeps all predictors. Lasso penalizes absolute coefficients and can set some to zero. Elastic net mixes both penalties and can retain groups of correlated predictors.
Scaling matters because penalties act on coefficient size. Tune penalty strength using a validation plan; for elastic net, also choose the mixture. Repeat preprocessing and selection inside resampling to avoid optimistic results.
Firth logistic regression changes the likelihood to reduce first-order bias. It can give finite estimates under separation. Its information-based penalty differs from ridge, lasso and elastic net. Do not treat a finite estimate as precise or as a substitute for adequate data.
Worked example
A worked example
- Setting
- A toy linear model has one centered predictor.
- Question
- What does ridge shrinkage do to a slope?
- Approach
- The predictor sum of squares is 1. Its ordinary slope is 2. Minimize squared error plus lambda times squared slope, with lambda equal to 1.
- Result
- The ridge slope falls from 2 to 1, a change of -1. The formula is 2 divided by (1 + 1).
- What it means
- This fixed-penalty example shows shrinkage only. It does not choose the tuning value or validate a prediction model.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
Firth bias reduction and prediction penalties solve different problems. None removes confounding or makes selected coefficients causal.
- Choosing the penalty on the final test set.Instead: Tune inside training resamples. Keep a separate test set, or use nested resampling for an honest assessment.
- Treating Firth as a variable-selection method.Instead: Firth reduces likelihood bias and handles separation. It is not lasso selection or a guarantee of better calibration.
Inside Mallard
How Mallard uses this
Mallard lists Choosing a penalty under Core research concepts.
concept.choosing_penalty, the id a plan cue records.Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Regularization Paths for Generalized Linear Models via Coordinate Descent (opens in a new tab) · J Stat Softw
- A solution to the problem of separation in logistic regression (opens in a new tab) · Stat Med · doi 10.1002/sim.1047