Ask Mallard

Prediction model validation

A validation study takes a prediction model that already exists and measures how well it performs on data it was not built from.
NHMRC EVIDENCE LEVEL · 2009Not yet classified · PrognosisProvisional design classification. Study quality and certainty of findings require separate appraisal.
Why this level?Specify whether cohort members are disease-free or at the same disease stage at inception. NHMRC 2009 source

Validation of future-outcome predictions uses prognosis criteria; validation of a contemporaneous diagnostic test uses diagnostic accuracy criteria. External validation is valuable, but the label alone does not establish a level.

What question does this help answer?

Not whether the model is a good model in the abstract, but whether its predicted probabilities can be trusted in THIS population, in this setting, at this time.

Development and validation are different studies

Developing a model chooses predictors and estimates coefficients. Validating one changes nothing: the model is applied exactly as published, coefficients and all, and its output is compared with what actually happened. A study that refits the coefficients on new data has developed a second model, not validated the first, and must report itself that way.

Internal, temporal, geographic, external

Internal validation (bootstrapping or cross-validation) corrects the optimism from having chosen the model on these same data. It is a necessary part of development, not an answer to the external question. Temporal validation applies the model to a later period, geographic validation to a different place, and full external validation to data collected independently by other investigators. They test progressively stronger claims.

Where validation data come from

Where validation data come from relative to the data a model was developed on. Internal validation resamples the development data itself. Temporal validation uses later data from the same setting. Geographic validation uses data from another setting in the same period. External validation uses later data from elsewhere.Development data, and the four places validation data can come fromWhere the model was builtDevelopment dataInternal: resampledTemporal: laterGeographic: elsewhereExternal: later,elsewhereTimeShaded windows are a different setting. A random split of the development data is internalvalidation and does not answer whether the model works elsewhere; calibration is what usually failswhen it moves.
Conceptual diagram. It shows the structure of the idea, not data from any study.

Calibration is usually what fails

Discrimination, the ability to rank a patient who has the outcome above one who does not, tends to travel reasonably well. Calibration, the agreement between predicted probability and observed frequency, often does not, because outcome prevalence differs between settings. A model with an unchanged c-statistic can still be systematically predicting 20 percent where the truth is 8 percent, and a c-statistic reported alone hides that entirely.

What to report

  • A calibration plot of observed against predicted risk across the whole range, not only a Hosmer-Lemeshow p-value.
  • Calibration-in-the-large (the intercept) and the calibration slope, which say whether the model is systematically off and whether its predictions are too extreme.
  • Discrimination, as a c-statistic or area under the ROC curve, with a confidence interval.
  • Clinical usefulness at the thresholds a decision would actually use, such as a decision curve.
  • The number of OUTCOME EVENTS, which drives precision far more than the number of participants.

Why Mallard may recommend it

Mallard may point here when a description says an existing score or model will be applied to a new cohort, or when the plan is to check whether a published risk tool works locally.

How it is sized

Validation is sized by events, not by a two-group difference, and the target is precision in the calibration slope and the c-statistic rather than power against a null. Mallard does not compute a headline sample size for a validation study from a closed form; it names the criteria and the events required instead of inventing a number.

What to watch out for

A model that performs well where it was built routinely performs worse elsewhere, and the usual failure is CALIBRATION rather than discrimination: the ranking survives, the absolute probabilities do not. Validating on a random split of the development data is internal validation and does not answer the external question.

If the model does not calibrate

A poorly calibrated model is not necessarily useless. Recalibrating the intercept, or the intercept and slope together, can restore it at far lower cost than building a new model, and the updated version then needs validating in its turn. Say plainly that the reported model is an updated one.

Clinical example

Question: Does a published pneumonia mortality score work in our hospital? Design: Apply the published coefficients unchanged to consecutive admissions, and report a calibration plot, calibration intercept and slope, the c-statistic and a decision curve at the thresholds the admission decision uses.

Recent studies that used this
Searching PubMed…
Part of the learning path Build and check a prediction model (step 5 of 5).
Related Learn pages
Sources & further reading
  1. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement (opens in a new tab) · BMJ · doi 10.1136/bmj.g7594
  2. Towards better clinical prediction models: seven steps for development and an ABCD for validation (opens in a new tab) · Eur Heart J · doi 10.1093/eurheartj/ehu207
  3. Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes (opens in a new tab) · Stat Med · doi 10.1002/sim.7992
TRIPOD is a reporting guideline: it says what a prediction model study must disclose, not how to design or analyse one. The second reference is the methodological companion, and the third is where the sample size reasoning comes from.
Last source review: 2026-09-06
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.