Ask Mallard

Prediction leakage, overfitting and optimism

Keep information boundaries intact when developing and evaluating a prediction model.

Three different problems

Leakage gives the modeling process information it should not have for the intended prediction task or evaluation. Overfitting learns chance patterns in development data. Optimism is the tendency for performance measured through the development process to look better than it will on appropriate new data. They overlap, but they are not synonyms and do not have one universal correction.

Start with the moment of prediction

Define who will receive a prediction, when it is made, the outcome horizon and which inputs are available then. A discharge code, post-outcome test or treatment decision made after deterioration can make an admission model look impressive while answering an unusable question. Check variable timestamps and clinical workflows, not only column names.

Protect the evaluation boundary

  • Choose the split or resampling unit to reflect intended use. Keeping related records from one patient in both training and test sets can leak patient-specific information.
  • Learn scaling, imputation, feature selection and tuning within training data in each resample. Repeating only the final model fit understates the optimism of a larger selection process.
  • Use nested resampling when an outer evaluation needs protection from tuning decisions made in an inner loop.
  • Repeatedly consulting a holdout to choose models turns that holdout into development information. Track every such choice and obtain a suitable new evaluation.
  • A temporal or geographical split may better address a transport question, but it also changes case mix and context; name the generalization claim being assessed.

What a leaky and a protected pipeline look like

Two pipelines for evaluating a prediction model. In the leaky one, scaling, imputation, feature selection and tuning are fitted on all the data before it is split, so the test data has already shaped the model. In the protected one, the data is split first, every fitted step is learned on the training data only, and the test data is used only to score the finished model.Leaky: steps fitted before the splitAll the dataScaling, imputation, feature selection and tuning fitted on all of itSplit into training and test dataModel trainedTest data scores the modelProtected: split first, fit on training data onlyAll the dataSplit first, by the unit of intended useTraining dataTest data, held backScaling, imputation, selection and tuningfitted hereFinished model scored once on the testdataWith resampling, the same holds inside every fold: repeating only the final fit understates optimism.
Conceptual diagram. It shows the structure of the idea, not data from any study.

Assess and reduce optimism

Appropriate bootstrap validation or cross-validation evaluates the full development procedure. Prespecified complexity limits and penalization can reduce overfitting, but tuning them is part of that procedure. External validation evaluates a fixed model in new data. If the model is updated there, describe the update and evaluate the updated version separately.

Example

A deterioration model includes a test ordered after a clinician already suspects deterioration. Moving preprocessing inside cross-validation does not fix this timing leak. Removing the unavailable information and redefining the prediction moment is necessary before resampling can evaluate the intended task.

Record in the workspace

Record prediction time, input availability, participant/group split rules, preprocessing and tuning sequence, evaluation-data access, model version and the planned performance measures. Keep apparent, internally validated and external performance clearly labelled.

What to watch out for

An untouched test label is not enough if evaluation information has already influenced features, preprocessing, tuning or cohort selection. Internal validation cannot repair predictors unavailable at the intended prediction time.

Recent studies that used this
Searching PubMed…
Part of the learning path Build and check a prediction model (step 2 of 5).
Related Learn pages
Sources & further reading
  1. Leakage and the reproducibility crisis in machine-learning-based science (opens in a new tab) · Patterns · doi 10.1016/j.patter.2023.100804
  2. Evaluation of clinical prediction models (part 1): from development to external validation (opens in a new tab) · BMJ · doi 10.1136/bmj-2023-074819
  3. Evaluation of clinical prediction models (part 2): how to undertake an external validation study (opens in a new tab) · BMJ · doi 10.1136/bmj-2023-074820
Prepared with AI assistance. Methodological and source checks are documented in this release; this is educational material, not independent professional approval. Reporting guidance supports transparent reporting and does not establish validity.
Last source review: 2026-09-09
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.