Ask Mallard
Core concept

Choosing a penalty

5 min read

Choose ridge, lasso or elastic net for a prediction goal; consider Firth logistic regression for small-sample bias or separation.

Fit

When it matters, and when it does not

It matters when

  • You need to limit unstable coefficients when building a prediction model.
  • Your logistic model has separation: some predictors perfectly distinguish outcomes and ordinary estimates can diverge.

It matters less when

Before you start

What to check in your study

Decide these first
  1. The problem to solveDistinguish predictive accuracy, a compact model and finite logistic estimates.
  2. Predictor scalesChoose scaling before applying a penalty. Handle scaling within each training sample.
  3. How tuning will be assessedKeep test data out of fitting, feature selection and tuning.
It rests on these
  1. Validation reflects how the model will be used.How to check: Choose resampling units that preserve clustering and time order when needed.
  2. A finite estimate still needs enough information.How to check: For Firth fits, inspect uncertainty and prefer appropriate penalized likelihood inference over a claim of certainty.

Plain language

How it works

Ridge penalizes squared coefficients and usually keeps all predictors. Lasso penalizes absolute coefficients and can set some to zero. Elastic net mixes both penalties and can retain groups of correlated predictors.

The choice at a glance
Prediction with all predictors: Shrink squared coefficients leads to Ridge; Sparse prediction: Allow coefficients to become zero leads to Lasso; Mixed shrinkage: Combine squared and absolute penalties leads to Elastic net; Logistic separation or bias: Information-based likelihood penalty leads to Firth logistic regression.Prediction with all predictorsShrink squared coefficientsRidgeUse the study question and assumptions together.Sparse predictionAllow coefficients to become zeroLassoUse the study question and assumptions together.Mixed shrinkageCombine squared and absolute penaltiesElastic netUse the study question and assumptions together.Logistic separation or biasInformation-based likelihood penaltyFirth logistic regressionUse the study question and assumptions together.
Conceptual diagram. It shows the structure of the idea, not data from any study.
Reading the choice

Scaling matters because penalties act on coefficient size. Tune penalty strength using a validation plan; for elastic net, also choose the mixture. Repeat preprocessing and selection inside resampling to avoid optimistic results.

What decides it

Firth logistic regression changes the likelihood to reduce first-order bias. It can give finite estimates under separation. Its information-based penalty differs from ridge, lasso and elastic net. Do not treat a finite estimate as precise or as a substitute for adequate data.

Worked example

A worked example

Invented for illustration
Setting
A toy linear model has one centered predictor.
Question
What does ridge shrinkage do to a slope?
Approach
The predictor sum of squares is 1. Its ordinary slope is 2. Minimize squared error plus lambda times squared slope, with lambda equal to 1.
Result
The ridge slope falls from 2 to 1, a change of -1. The formula is 2 divided by (1 + 1).
What it means
This fixed-penalty example shows shrinkage only. It does not choose the tuning value or validate a prediction model.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

Firth bias reduction and prediction penalties solve different problems. None removes confounding or makes selected coefficients causal.

  • Choosing the penalty on the final test set.Instead: Tune inside training resamples. Keep a separate test set, or use nested resampling for an honest assessment.
  • Treating Firth as a variable-selection method.Instead: Firth reduces likelihood bias and handles separation. It is not lasso selection or a guarantee of better calibration.

Inside Mallard

How Mallard uses this

Mallard lists Choosing a penalty under Core research concepts.

No Mallard code example points to this page.
Atlas node concept.choosing_penalty, the id a plan cue records.
Part of the learning path Build and check a prediction model (step 3 of 5).

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. Regularization Paths for Generalized Linear Models via Coordinate Descent (opens in a new tab) · J Stat Softw
  2. A solution to the problem of separation in logistic regression (opens in a new tab) · Stat Med · doi 10.1002/sim.1047
Prepared with AI assistance. Source titles, authors, publication years and available DOIs were verified against PubMed on 2026-11-06. No independent professional approval is claimed. Friedman and colleagues develop estimation paths for ridge, lasso and elastic net. Heinze and Schemper demonstrate Firth logistic regression under separation and compare inference procedures. PubMed lists no DOI for the coordinate-descent record.
Last source review: 2026-11-06
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.