Calibration and discrimination
Why the distinction matters clinically
If a model is used to rank patients for a fixed number of intensive-care beds, discrimination is what matters. If it is used to tell a patient their risk is 8 percent, calibration is what matters, and getting it wrong changes real decisions in a way a good c-statistic will not reveal.
What miscalibration looks like
The four levels of calibration
- Mean calibration, or calibration-in-the-large: does the average predicted risk match the observed event rate? The weakest requirement and the one that fails first in a new setting.
- Weak calibration: is the calibration slope 1? A slope below 1 means predictions are too extreme, the signature of overfitting.
- Moderate calibration: does observed risk match predicted risk across the whole range? This is what a calibration curve shows.
- Strong calibration: correct for every combination of covariates. Unattainable in practice and not a useful target.
- Report at least the intercept, the slope and a smoothed calibration curve.
Why calibration travels badly
Discrimination depends on the separation between groups, which is often similar across settings. Calibration depends on the absolute event rate, which varies enormously between a specialist referral centre and primary care. A model transported without recalibration will frequently be well ordered and badly wrong about magnitude.
Clinical usefulness is a third question
Even a well-calibrated model is only useful if acting on it beats the alternatives at the thresholds a clinician would actually use. Decision curve analysis answers that by computing net benefit across a range of thresholds, weighting false positives against false negatives the way the decision does.
What to watch out for
Discrimination is reported almost universally and calibration almost never, which is the wrong way round for a model that quotes a risk to a patient. A model can rank perfectly and be systematically wrong about absolute risk, and the c-statistic will not budge. The Hosmer-Lemeshow test is not an adequate substitute for a calibration plot: it has almost no power in a small sample and rejects trivial miscalibration in a large one, so its p-value is mostly a function of sample size.
Clinical example
A published cardiovascular risk score is applied in a new population. Discrimination is unchanged, but the calibration curve shows predicted 10 percent risks corresponding to 4 percent observed. The ranking is fine; the numbers quoted to patients are more than double the truth, and the model needs recalibrating before use.
- Calibration: the Achilles heel of predictive analytics (opens in a new tab) · BMC Med · doi 10.1186/s12916-019-1466-7
- Towards better clinical prediction models: seven steps for development and an ABCD for validation (opens in a new tab) · Eur Heart J · doi 10.1093/eurheartj/ehu207