Bootstrap internal validation
Bootstrap internal validation estimates how much a fitted prediction model flatters itself when tested on its own development data.
Fit
When to use it, and when not to
Use it when
- You develop a prediction model and want to estimate its performance in similar people.
- You can repeat the whole development procedure in each resample.
Not when
- You need evidence from a new setting. Use an external validation study. Read: Prediction model validation
- You only need an interval for a fixed statistic. Use a bootstrap confidence interval. Read: The bootstrap
Before you start
What you need
- The complete development procedureInclude imputation, transformations, feature selection and tuning.
- A performance measure and its directionFor a measure where higher is better, subtract average optimism from apparent performance.
- The unit to resampleRespect clinics, matched sets and other dependence in the sampling design.
- All data-driven steps are repeated within each resample.How to check: Fit preprocessing on the resample, then apply that fitted procedure when evaluating the original data.
- The development sample represents the intended source population.How to check: Inspect recruitment and missing data. Resampling repeats selection bias; it cannot remove it.
- The procedure can be fitted reliably in resamples.How to check: Record failures and check that results settle as you add resamples.
Plain language
How it works
First fit the full development procedure on the original data and measure its apparent performance. Draw samples of the same size with replacement. Some people appear more than once.
Refit the whole procedure in each resample. Evaluate that fitted model on both its resample and the original data. Their difference estimates optimism for that resample.
Average those differences and subtract them from apparent performance when higher values are better. For a loss, keep the sign convention explicit. Steyerberg and colleagues compared validation procedures; Harrell and colleagues describe overfitting and resampling.
Worked example
A worked example
- Setting
- A team develops a model for readmission after discharge.
- Question
- How much does its apparent ranking ability overstate internal performance?
- Approach
- Use the area under the receiver operating characteristic curve to measure ranking. For three illustrative resamples, training scores are 0.84, 0.83 and 0.85. Original-data scores from those fitted models are 0.80, 0.79 and 0.81.
- Result
- Each gap is 0.04, giving average optimism of 0.04. Apparent performance of 0.82 becomes 0.78 after correction.
- What it means
- The corrected score estimates performance in similar new people. These three resamples teach the arithmetic; a real analysis needs many more.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
Repeat every data-driven development step within each bootstrap sample. Correcting optimism estimates internal performance; it does not establish performance in a new setting.
- Keeping variables selected on the full dataset fixed.Instead: Repeat selection within each resample, alongside fitting and tuning.
- Calling corrected performance external validation.Instead: Obtain new data from the intended setting before making that claim.
Inside Mallard
How Mallard uses this
The method registry links this method to its published analysis article and code. Mallard does not run this analysis.
penalized-prediction-optimism).analysis.bootstrap_internal_validation, the id a plan cue records.Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Internal validation of predictive models: efficiency of some procedures for logistic regression analysis (opens in a new tab) · J Clin Epidemiol · doi 10.1016/s0895-4356(01)00341-9
- Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors (opens in a new tab) · Stat Med · doi 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4