Prediction model performance and validation
What question does this help answer?
For the patients and prediction time in this aim, how well does the model estimate a defined outcome over a defined horizon? A prediction question concerns what will happen; predictor coefficients do not establish what would happen if a treatment were changed.
When to use it
Plan evaluation during model development and when assessing an existing score in another place, period or patient group. State the intended decision, predictors available at that moment, outcome ascertainment and prediction horizon before choosing performance measures.
When not to use it
A validation dataset cannot evaluate the intended use if it lacks the required predictors, horizon or reliable outcomes. Do not substitute apparent training performance or a highly selected convenience sample for an assessment of the claimed setting.
Three different performance questions
- Discrimination: can the model distinguish people with different outcomes? Select an AUC or concordance measure appropriate to the outcome and horizon.
- Calibration: do predicted risks agree with observed outcomes? Examine a calibration curve and appropriate numerical summaries, with uncertainty; average agreement alone can conceal important errors.
- Usefulness: would a decision based on predictions improve the tradeoff between benefits and harms? Consider net benefit across clinically defensible thresholds and compare with relevant alternatives.
Prevent leakage before measuring optimism
Leakage occurs when information unavailable at prediction time, or information from evaluation data, enters model development. Keep a patient's related records together when independence across patients is intended. Learn imputation, scaling, predictor selection and tuning inside each training fold or resample. Outcome-informed choices made before splitting can contaminate an apparently held-out test set.
Match validation to the claim
Bootstrap validation or appropriately designed cross-validation can estimate internal optimism when the entire model-building process is repeated. External evaluation applies a fixed model to new data and examines performance in the named setting. If that dataset is used to update coefficients or thresholds, report the updating separately and evaluate the updated model with an appropriate new validation strategy.
Key assumptions and design requirements
- Plan evaluation sample size for precision of the chosen performance measures; no single events-per-variable rule establishes validation adequacy.
- Use methods appropriate to censoring and competing events for time-to-event predictions.
- Record missing-data assumptions and how predictors will be handled in actual use.
- Report uncertainty, relevant subgroup performance and differences in case mix or care; successful validation is not a permanent safety certificate.
Clinical example
An admission score predicts deterioration within 48 hours. Adding a laboratory result ordered after deterioration would leak future information. The team instead fixes admission-time inputs, evaluates the complete development procedure, then assesses the fixed score in a later hospital cohort. They examine calibration and useful decision thresholds alongside discrimination; this example supplies no invented performance results.
What to watch out for
Performance measured on the data used to choose predictors, tune the model or set thresholds is optimistic. A good AUC does not establish calibration, clinical usefulness or safe performance in another population.
Before using this in your workspace
Keep development and evaluation decisions attached to their respective aims. Record predictor availability, resampling unit, model version, evaluation population and planned performance measures before reviewing results.
- Evaluation of clinical prediction models (part 1): from development to external validation (opens in a new tab) · BMJ · doi 10.1136/bmj-2023-074819
- Evaluation of clinical prediction models (part 2): how to undertake an external validation study (opens in a new tab) · BMJ · doi 10.1136/bmj-2023-074820
- Don't be misled: 3 misconceptions about external validation of clinical prediction models (opens in a new tab) · J Clin Epidemiol · doi 10.1016/j.jclinepi.2024.111387