Prediction model performance and validation
Evaluating a prediction model asks whether its predicted risks are accurate and useful for the patients, setting and decision it is meant for.
Fit
When it matters, and when it does not
It matters when
- You have built a model and need an honest estimate of how it will perform on new patients.
- You want to use an existing score in another hospital, period or patient group.
- A decision will rest on the predicted risk, such as who gets an early review.
It matters less when
- Your question is whether changing a factor would change the outcome. That is a causal question, and a prediction model cannot answer it. Read: From research question to estimand
- The model is not built yet. Plan its development first: its predictors, its sample size and how it will be fitted. Read: Prediction model development
- The validation data lack the model's predictors, its time horizon or reliable outcomes. Such data cannot test the intended use.
Before you start
What to check in your study
- The intended usetargetWho is predicted, at what moment, which outcome over what horizon, and which decision follows.
- Predictors available at the moment of predictionAnything measured later leaks the future into the model and makes it look better than it is.
- How performance will be estimatedInternal validation checks the development procedure. External validation checks a fixed model in a new setting.
- The performance measures, named in advancesummaryMeasureDiscrimination, calibration and net benefit at thresholds that make clinical sense.
- Enough events for precise estimatesSize the validation for the precision of calibration and discrimination. No events-per-variable rule settles it.
- No information from the evaluation data, or from after the prediction moment, shaped the model.How to check: Fit imputation, scaling, predictor selection and tuning inside each training sample. Keep each patient's records on one side of a split.
- The evaluation patients resemble those the model will be used on.How to check: Compare case mix, outcome rates and care with the intended setting. Performance can drift as any of them change.
- Missing predictors are handled in evaluation as they will be in use.How to check: State the missing-data assumption, and what happens when a real patient's value is missing.
- Outcomes are recorded the same way for everyone, over the full horizon.How to check: For time-to-event predictions, handle censoring, and competing events such as death, with methods built for them.
Plain language
How it works
A prediction model needs more than one kind of check. Ranking, risk accuracy and clinical usefulness answer different questions. A good result for one does not settle the others. Use the ROC page for discrimination, calibration assessment for risk accuracy, and decision curve analysis for the trade-off at decision thresholds.
Internal validation estimates how development choices affect performance in similar new patients. Bootstrap internal validation teaches that procedure. External validation asks whether a fixed model works in another setting. Keep updating separate from evaluation. The development and validation design pages cover who enters each study and what must be planned.
Worked example
A worked example
- Setting
- A deterioration score developed elsewhere is applied, unchanged, to 1,000 admissions at a new hospital.
- Question
- Are its predicted risks right here, and would acting on them help?
- Approach
- Apply the fixed model before any updating. Plan separate checks of ranking, risk accuracy and the value of acting on predictions.
- Result
- There were 150 of 1,000 events, or 15 in 100. The mean predicted risk was 10 in 100.
- What it means
- These averages flag a mismatch, not its shape. Calibration assessment checks risk accuracy; a decision curve addresses usefulness at chosen thresholds.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
Performance measured on the data used to choose predictors, tune the model or set thresholds is optimistic. A good AUC does not establish calibration, clinical usefulness or safe performance in another population.
- Treating a high area under the curve as proof that risk estimates or decisions are sound.Instead: Assess ranking, calibration and usefulness as separate questions. Choose the relevant single-method assessments.
- Sizing a model's development by 10 events per predictor.Instead: That is a floor, not a calculation. The Riley criteria size on the outcome rate, expected performance, predictor parameters and shrinkage.
- Re-estimating coefficients or thresholds in the new data and calling it external validation.Instead: That is updating. Report it separately, and validate the updated model again.
- Treating an internal check as evidence that a model works in another setting.Instead: Internal and external validation address different claims. State which population the evaluation represents.
Inside Mallard
How Mallard uses this
When your plan's target is prediction or prognosis, Mallard's method list may include checks of discrimination and calibration, decision curve analysis, or external validation of a survival model. Plans do not link to it yet.
concept.prediction_performance, the id a plan cue records.Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Evaluation of clinical prediction models (part 1): from development to external validation (opens in a new tab) · BMJ · doi 10.1136/bmj-2023-074819
- Evaluation of clinical prediction models (part 2): how to undertake an external validation study (opens in a new tab) · BMJ · doi 10.1136/bmj-2023-074820
- Don't be misled: 3 misconceptions about external validation of clinical prediction models (opens in a new tab) · J Clin Epidemiol · doi 10.1016/j.jclinepi.2024.111387
- Decision curve analysis: a novel method for evaluating prediction models (opens in a new tab) · Med Decis Making · doi 10.1177/0272989X06295361