Ask Mallard
Analysis method

Calibration assessment

4 min read

Calibration assessment checks whether predicted risks match observed outcomes, across the risk range and at the horizon you intend to use.

Fit

When to use it, and when not to

Use it when

  • You evaluate a model that gives individual risks for a defined outcome and horizon.
  • You want to find where predictions are too high or too low.

Not when

Before you start

What you need

Decide these first
  1. Fixed model predictions and observed outcomesUse predictions made without learning from the evaluation outcomes.
  2. A defined outcome and horizonUse methods that handle censoring when outcomes are not known for everyone.
  3. Enough events and nonevents across the risk rangePlan precision for the curve, intercept and slope; tail estimates may be unstable.
It rests on these
  1. Observed outcomes match the prediction target.How to check: Check outcome definitions, follow-up and losses before comparing risks.
  2. Evaluation data have not shaped model development.How to check: Use external data or an internal validation procedure that repeats development.
  3. The chosen smoother can reveal meaningful errors.How to check: Show uncertainty and the risk distribution. Do not rely on one test or grouped points alone.

Plain language

How it works

Plot observed risk against predicted risk. The ideal line has equal values on both axes. A smooth curve can show local errors that an overall average hides.

Reading the method

For binary outcomes, fit outcome on the log odds of predicted risk. The ideal intercept is zero and slope is one. Estimate calibration-in-the-large with the slope fixed at one.

Scope and evidence

A slope below one often indicates predictions that are too extreme. It does not diagnose a unique cause. Van Calster and colleagues describe calibration levels. A simulation by van den Goorbergh and colleagues shows how class balancing can distort risks without improving ranking.

The method in outline
Compare risks at the same horizon. An average or slope can hide local errors. The curve and summaries answer complementary calibration questions.Compare risks at the same horizonFixed predictionsOutcomes at the stated horizonCurve across the risk rangeIntercept and slope summariesAssess errors and uncertaintyAn average or slope can hide local errors. The curve and summaries answer complementary calibrationquestions.
Conceptual diagram. It shows the structure of the idea, not data from any study.

Worked example

A worked example

Invented for illustration
Setting
A hospital evaluates fixed predictions of readmission by a stated follow-up horizon.
Question
Do risks agree with outcomes in the patients assigned a similar risk?
Approach
A risk band contains 100 patients with mean predicted risk of 20 in 100. All outcomes are observed.
Result
There are 16 of 100 events, or 16 in 100. Predicted risk exceeds observed risk by 4 percentage points in this band.
What it means
That band suggests overprediction. It cannot establish calibration elsewhere or supply a fitted slope.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

Good average calibration can hide large errors across the risk range. Use a curve and uncertainty; neither a good slope nor a large p-value proves accurate risks.

  • Treating a nonsignificant goodness-of-fit test as proof.Instead: Inspect a smooth calibration curve with uncertainty.
  • Recalibrating and assessing the update on the same outcomes.Instead: Evaluate the updated model with a fresh validation procedure.

Inside Mallard

How Mallard uses this

The method registry links this method to its published analysis article and code. Mallard does not run this analysis.

Code example in Mallard: Calibration of a prediction model: slope, calibration-in-the-large and O/E (calibration-slope-intercept).
Atlas node analysis.calibration_assessment, the id a plan cue records.
Code for this method
Loading…

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. A calibration hierarchy for risk models was defined: from utopia to empirical data (opens in a new tab) · J Clin Epidemiol · doi 10.1016/j.jclinepi.2015.12.005
  2. Calibration: the Achilles heel of predictive analytics (opens in a new tab) · BMC Med · doi 10.1186/s12916-019-1466-7
  3. The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression (opens in a new tab) · J Am Med Inform Assoc · doi 10.1093/jamia/ocac093
Prepared with AI assistance. PubMed records were checked by PMID, title, authors, year and DOI on 2026-11-06. The calibration hierarchy supports the distinction between average, slope and curve assessment. The 2019 guidance supports practical evaluation. The class-imbalance simulation supports the warning that ranking and risk accuracy can diverge. No independent professional approval is claimed.
Last source review: 2026-11-06
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.