Ask Mallard
Core concept

Prediction model performance and validation

4 min read

Evaluating a prediction model asks whether its predicted risks are accurate and useful for the patients, setting and decision it is meant for.

Fit

When it matters, and when it does not

It matters when

  • You have built a model and need an honest estimate of how it will perform on new patients.
  • You want to use an existing score in another hospital, period or patient group.
  • A decision will rest on the predicted risk, such as who gets an early review.

It matters less when

  • Your question is whether changing a factor would change the outcome. That is a causal question, and a prediction model cannot answer it. Read: From research question to estimand
  • The model is not built yet. Plan its development first: its predictors, its sample size and how it will be fitted. Read: Prediction model development
  • The validation data lack the model's predictors, its time horizon or reliable outcomes. Such data cannot test the intended use.

Before you start

What to check in your study

Decide these first
  1. The intended usetargetWho is predicted, at what moment, which outcome over what horizon, and which decision follows.
  2. Predictors available at the moment of predictionAnything measured later leaks the future into the model and makes it look better than it is.
  3. How performance will be estimatedInternal validation checks the development procedure. External validation checks a fixed model in a new setting.
  4. The performance measures, named in advancesummaryMeasureDiscrimination, calibration and net benefit at thresholds that make clinical sense.
  5. Enough events for precise estimatesSize the validation for the precision of calibration and discrimination. No events-per-variable rule settles it.
It rests on these
  1. No information from the evaluation data, or from after the prediction moment, shaped the model.How to check: Fit imputation, scaling, predictor selection and tuning inside each training sample. Keep each patient's records on one side of a split.
  2. The evaluation patients resemble those the model will be used on.How to check: Compare case mix, outcome rates and care with the intended setting. Performance can drift as any of them change.
  3. Missing predictors are handled in evaluation as they will be in use.How to check: State the missing-data assumption, and what happens when a real patient's value is missing.
  4. Outcomes are recorded the same way for everyone, over the full horizon.How to check: For time-to-event predictions, handle censoring, and competing events such as death, with methods built for them.

Plain language

How it works

A prediction model needs more than one kind of check. Ranking, risk accuracy and clinical usefulness answer different questions. A good result for one does not settle the others. Use the ROC page for discrimination, calibration assessment for risk accuracy, and decision curve analysis for the trade-off at decision thresholds.

What a calibration plot shows
A calibration plot of predicted risk against observed risk. A dashed diagonal shows perfect calibration. The model's curve lies below it everywhere, meaning it predicts risks that are too high.Calibration: predicted risk against the risk that actually occurredPredicted riskObserved riskPerfect calibrationThis model: predicts too highThe model can still rank people well, so its discrimination looks fine, while every absolute risk itquotes is too high.
Conceptual diagram. It shows the structure of the idea, not data from any study.
Choose the evaluation

Internal validation estimates how development choices affect performance in similar new patients. Bootstrap internal validation teaches that procedure. External validation asks whether a fixed model works in another setting. Keep updating separate from evaluation. The development and validation design pages cover who enters each study and what must be planned.

Worked example

A worked example

Invented for illustration
Setting
A deterioration score developed elsewhere is applied, unchanged, to 1,000 admissions at a new hospital.
Question
Are its predicted risks right here, and would acting on them help?
Approach
Apply the fixed model before any updating. Plan separate checks of ranking, risk accuracy and the value of acting on predictions.
Result
There were 150 of 1,000 events, or 15 in 100. The mean predicted risk was 10 in 100.
What it means
These averages flag a mismatch, not its shape. Calibration assessment checks risk accuracy; a decision curve addresses usefulness at chosen thresholds.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

Performance measured on the data used to choose predictors, tune the model or set thresholds is optimistic. A good AUC does not establish calibration, clinical usefulness or safe performance in another population.

  • Treating a high area under the curve as proof that risk estimates or decisions are sound.Instead: Assess ranking, calibration and usefulness as separate questions. Choose the relevant single-method assessments.
  • Sizing a model's development by 10 events per predictor.Instead: That is a floor, not a calculation. The Riley criteria size on the outcome rate, expected performance, predictor parameters and shrinkage.
  • Re-estimating coefficients or thresholds in the new data and calling it external validation.Instead: That is updating. Report it separately, and validate the updated model again.
  • Treating an internal check as evidence that a model works in another setting.Instead: Internal and external validation address different claims. State which population the evaluation represents.

Inside Mallard

How Mallard uses this

When your plan's target is prediction or prognosis, Mallard's method list may include checks of discrimination and calibration, decision curve analysis, or external validation of a survival model. Plans do not link to it yet.

No Mallard code example points to this page.
Atlas node concept.prediction_performance, the id a plan cue records.

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. Evaluation of clinical prediction models (part 1): from development to external validation (opens in a new tab) · BMJ · doi 10.1136/bmj-2023-074819
  2. Evaluation of clinical prediction models (part 2): how to undertake an external validation study (opens in a new tab) · BMJ · doi 10.1136/bmj-2023-074820
  3. Don't be misled: 3 misconceptions about external validation of clinical prediction models (opens in a new tab) · J Clin Epidemiol · doi 10.1016/j.jclinepi.2024.111387
  4. Decision curve analysis: a novel method for evaluating prediction models (opens in a new tab) · Med Decis Making · doi 10.1177/0272989X06295361
Prepared with AI assistance; source records were first resolved against PubMed by 2026-09-09, and each was rechecked by PMID, with its title matched, on 2026-11-05. Publication follows agent methodological review under the owner’s expansion request; no independent professional approval is claimed. Evaluation supports a stated use in a stated setting; it does not by itself demonstrate that deploying the model improves patient outcomes. Retained PubMed records were rechecked by PMID and title on 2026-11-06: record verified, confirm it supports this use.
Last source review: 2026-11-06
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.