Ridge regression
Ridge regression shrinks fitted slopes toward zero by penalizing their squared sizes, often stabilizing predictions when predictors overlap.
Fit
When to use it, and when not to
Use it when
- You want prediction and need to keep the candidate predictors.
- Predictors are strongly related and unpenalized slopes are unstable.
Not when
- You require a sparse model that sets slopes to zero. Consider a selection penalty. Read: Uniform shrinkage
- Your goal is a causal effect with a defined adjustment set. Use an analysis matched to that estimand. Read: The estimand
Before you start
What you need
- Outcome, prediction time and intended useoutcomeTypeChoose continuous or binary outcomes before the loss function. These worked examples use squared-error loss.
- Predictor coding, scaling and missing-data planFit scaling and imputation on training data only. Under MAR, assess imputation and include a sensitivity analysis if MNAR is plausible.
- Candidate parameters and validation planPlan the full sample for prediction development. Riley 2019 separates outcome types. Van Smeden shows why events per variable (EPV) is inadequate.
- The scaling and penalty convention are stated.How to check: The same numeric penalty can mean different shrinkage when software divides the loss by sample size.
- Training data cover the intended prediction setting.How to check: Check who was included and which predictors are available at prediction time.
- Performance assessment includes every fitted step.How to check: Fit imputation, scaling and penalty tuning within training folds. Use an outer assessment or bootstrap validation for the full procedure.
Plain language
How it works
For a continuous outcome, ridge minimizes squared prediction errors plus a penalty on squared slopes. The intercept is usually left unpenalized. Scale predictors first so units do not decide the penalty. Choose the strength using training data, and assess the full procedure on data kept outside that tuning.
Friedman and colleagues describe ridge within penalized model fitting. Positive ridge penalties generally shrink slopes without selecting a subset. A slope can still be zero because of the data. Riley gives outcome-specific prediction-development criteria; van Smeden shows that an EPV rule cannot establish adequate binary prediction performance.
Worked example
A worked example
- Setting
- A centered continuous outcome is modeled with one centered predictor. The predictor has sum of squares 1 and cross-product with the outcome 2.
- Question
- How does a fixed ridge penalty change the slope?
- Approach
- Minimize half the residual sum of squares plus lambda times half the squared slope. Fix lambda at 1; leave the intercept unpenalized.
- Result
- The unpenalized slope is 2. Ridge gives 1, a change of -1. At centered predictor value 2, the fitted centered outcome is 2.
- What it means
- The slope is smaller but remains in the model. These chosen inputs illustrate shrinkage; they do not show better prediction in new people.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
Shrinking coefficients can reduce prediction error, but cannot repair confounding, leakage or a small development sample. Ordinary unpenalized Wald intervals do not become valid ridge intervals merely by using the shrunken coefficients.
- Treating events per variable as the sample-size method.Instead: Events per variable (EPV) is a minimum floor, not adequate planning. Use outcome-specific prediction criteria or simulation, including all candidate parameters.
- Reporting ordinary unpenalized Wald intervals for penalized slopes.Instead: Coefficient inference needs a justified penalty-aware profile-likelihood or Wald method. Ordinary intervals after tuning or selection ignore that process.
- Selecting confounders by their predictive strength.Instead: Choose causal adjustment from the question and a causal model. Prediction shrinkage does not identify causal effects.
Inside Mallard
How Mallard uses this
Plans do not link to this method yet. No method or sizing entry is mapped to this new article.
analysis.ridge_regression, the id a plan cue records.Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Minimum sample size for developing a multivariable prediction model: Part I - Continuous outcomes (opens in a new tab) · Stat Med · doi 10.1002/sim.7993
- Regularization Paths for Generalized Linear Models via Coordinate Descent (opens in a new tab) · J Stat Softw
- Elastic Net Regularization Paths for All Generalized Linear Models (opens in a new tab) · J Stat Softw · doi 10.18637/jss.v106.i01
- Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes (opens in a new tab) · Stat Med · doi 10.1002/sim.7992
- Sample size for binary logistic prediction models: Beyond events per variable criteria (opens in a new tab) · Stat Methods Med Res · doi 10.1177/0962280218784726