Calibration assessment
Calibration assessment checks whether predicted risks match observed outcomes, across the risk range and at the horizon you intend to use.
Fit
When to use it, and when not to
Use it when
- You evaluate a model that gives individual risks for a defined outcome and horizon.
- You want to find where predictions are too high or too low.
Not when
- You only want to rank people by risk. Assess discrimination with a receiver operating characteristic curve. Read: ROC curves and diagnostic thresholds
- You need to choose an action from risks. Assess usefulness as well as calibration. Read: Prediction model performance and validation
Before you start
What you need
- Fixed model predictions and observed outcomesUse predictions made without learning from the evaluation outcomes.
- A defined outcome and horizonUse methods that handle censoring when outcomes are not known for everyone.
- Enough events and nonevents across the risk rangePlan precision for the curve, intercept and slope; tail estimates may be unstable.
- Observed outcomes match the prediction target.How to check: Check outcome definitions, follow-up and losses before comparing risks.
- Evaluation data have not shaped model development.How to check: Use external data or an internal validation procedure that repeats development.
- The chosen smoother can reveal meaningful errors.How to check: Show uncertainty and the risk distribution. Do not rely on one test or grouped points alone.
Plain language
How it works
Plot observed risk against predicted risk. The ideal line has equal values on both axes. A smooth curve can show local errors that an overall average hides.
For binary outcomes, fit outcome on the log odds of predicted risk. The ideal intercept is zero and slope is one. Estimate calibration-in-the-large with the slope fixed at one.
A slope below one often indicates predictions that are too extreme. It does not diagnose a unique cause. Van Calster and colleagues describe calibration levels. A simulation by van den Goorbergh and colleagues shows how class balancing can distort risks without improving ranking.
Worked example
A worked example
- Setting
- A hospital evaluates fixed predictions of readmission by a stated follow-up horizon.
- Question
- Do risks agree with outcomes in the patients assigned a similar risk?
- Approach
- A risk band contains 100 patients with mean predicted risk of 20 in 100. All outcomes are observed.
- Result
- There are 16 of 100 events, or 16 in 100. Predicted risk exceeds observed risk by 4 percentage points in this band.
- What it means
- That band suggests overprediction. It cannot establish calibration elsewhere or supply a fitted slope.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
Good average calibration can hide large errors across the risk range. Use a curve and uncertainty; neither a good slope nor a large p-value proves accurate risks.
- Treating a nonsignificant goodness-of-fit test as proof.Instead: Inspect a smooth calibration curve with uncertainty.
- Recalibrating and assessing the update on the same outcomes.Instead: Evaluate the updated model with a fresh validation procedure.
Inside Mallard
How Mallard uses this
The method registry links this method to its published analysis article and code. Mallard does not run this analysis.
calibration-slope-intercept).analysis.calibration_assessment, the id a plan cue records.Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- A calibration hierarchy for risk models was defined: from utopia to empirical data (opens in a new tab) · J Clin Epidemiol · doi 10.1016/j.jclinepi.2015.12.005
- Calibration: the Achilles heel of predictive analytics (opens in a new tab) · BMC Med · doi 10.1186/s12916-019-1466-7
- The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression (opens in a new tab) · J Am Med Inform Assoc · doi 10.1093/jamia/ocac093