Ask Mallard
Analysis method

Bootstrap internal validation

4 min read

Bootstrap internal validation estimates how much a fitted prediction model flatters itself when tested on its own development data.

Fit

When to use it, and when not to

Use it when

  • You develop a prediction model and want to estimate its performance in similar people.
  • You can repeat the whole development procedure in each resample.

Not when

Before you start

What you need

Decide these first
  1. The complete development procedureInclude imputation, transformations, feature selection and tuning.
  2. A performance measure and its directionFor a measure where higher is better, subtract average optimism from apparent performance.
  3. The unit to resampleRespect clinics, matched sets and other dependence in the sampling design.
It rests on these
  1. All data-driven steps are repeated within each resample.How to check: Fit preprocessing on the resample, then apply that fitted procedure when evaluating the original data.
  2. The development sample represents the intended source population.How to check: Inspect recruitment and missing data. Resampling repeats selection bias; it cannot remove it.
  3. The procedure can be fitted reliably in resamples.How to check: Record failures and check that results settle as you add resamples.

Plain language

How it works

First fit the full development procedure on the original data and measure its apparent performance. Draw samples of the same size with replacement. Some people appear more than once.

Reading the method

Refit the whole procedure in each resample. Evaluate that fitted model on both its resample and the original data. Their difference estimates optimism for that resample.

Scope and evidence

Average those differences and subtract them from apparent performance when higher values are better. For a loss, keep the sign convention explicit. Steyerberg and colleagues compared validation procedures; Harrell and colleagues describe overfitting and resampling.

The method in outline
Repeat development before comparing performance. Both evaluations use the same fitted resample model. Correct apparent performance for the average gap.Repeat development before comparing performanceOriginal development dataDraw a bootstrap sampleRepeat the whole development procedureEvaluate on bootstrap sampleEvaluate on original dataAverage paired performance gapsBoth evaluations use the same fitted resample model. Correct apparent performance for the average gap.
Conceptual diagram. It shows the structure of the idea, not data from any study.

Worked example

A worked example

Invented for illustration
Setting
A team develops a model for readmission after discharge.
Question
How much does its apparent ranking ability overstate internal performance?
Approach
Use the area under the receiver operating characteristic curve to measure ranking. For three illustrative resamples, training scores are 0.84, 0.83 and 0.85. Original-data scores from those fitted models are 0.80, 0.79 and 0.81.
Result
Each gap is 0.04, giving average optimism of 0.04. Apparent performance of 0.82 becomes 0.78 after correction.
What it means
The corrected score estimates performance in similar new people. These three resamples teach the arithmetic; a real analysis needs many more.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

Repeat every data-driven development step within each bootstrap sample. Correcting optimism estimates internal performance; it does not establish performance in a new setting.

  • Keeping variables selected on the full dataset fixed.Instead: Repeat selection within each resample, alongside fitting and tuning.
  • Calling corrected performance external validation.Instead: Obtain new data from the intended setting before making that claim.

Inside Mallard

How Mallard uses this

The method registry links this method to its published analysis article and code. Mallard does not run this analysis.

Code example in Mallard: Penalized prediction model development with bootstrap optimism correction (penalized-prediction-optimism).
Atlas node analysis.bootstrap_internal_validation, the id a plan cue records.
Code for this method
Loading…

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis (opens in a new tab) · J Clin Epidemiol · doi 10.1016/s0895-4356(01)00341-9
  2. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors (opens in a new tab) · Stat Med · doi 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4
Prepared with AI assistance. PubMed records were checked by PMID, title, authors, year and DOI on 2026-11-06. Steyerberg and colleagues compare internal validation procedures. Harrell and colleagues support the overfitting and whole-procedure resampling discussion. No independent professional approval is claimed.
Last source review: 2026-11-06
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.