Prediction model development
Why this level?
Specify whether cohort members are disease-free or at the same disease stage at inception. NHMRC 2009 sourceFuture-outcome prediction may be assessed under prognosis, but contemporaneous diagnosis uses diagnostic accuracy. Development alone does not establish a level: timing, inception and sampling must be specified.
What question does this help answer?
It answers "what is the chance this happens to this person", which is a different question from "what would happen if we changed this exposure". A predictor earns its place by being informative, not by being a cause.
When to use it
Use it when a clinical decision depends on an individual's risk and no adequate model exists, when the predictors are available at the point of care, and when enough outcome events exist to fit a model without overfitting it.
When not to use it
Do not use a prediction framing for a causal question. Coefficients from a prediction model are not causal effects and must not be reported as if they were: a variable can improve prediction while being a consequence of the outcome rather than a cause of it. Also avoid developing a new model when a validated one exists and could be evaluated instead.
How it works in plain language
Candidate predictors are specified, a model is fitted, and its performance is assessed on two axes that answer different questions: DISCRIMINATION, whether it ranks higher-risk people above lower-risk ones, and CALIBRATION, whether the predicted probabilities match the observed frequencies. A model can discriminate well and still be badly calibrated, which makes its numbers unusable at the bedside even though its c-statistic looks respectable.
Why Mallard may recommend it
Mallard may recommend prediction model development when the stated goal is to identify who is at risk rather than to estimate the effect of an exposure, and when the investigator describes building a score or rule.
Key assumptions and design requirements
- Sample size is justified by the Riley criteria rather than a rule of thumb such as ten events per variable.
- Predictors are all available at the time the prediction would actually be made.
- Both discrimination AND calibration are reported; a c-statistic alone is not performance.
- Apparent performance is optimistic, so internal validation (bootstrapping or cross-validation) is planned from the start.
- Selection of predictors by univariable screening is avoided, because it biases the model and its performance estimates.
What to watch out for
Prediction and explanation are different goals and a good model for one may be poor for the other: a predictor can earn its place by being informative without being a cause. Apparent performance measured on the development data is optimistic, so discrimination AND calibration must be reported, with internal validation at minimum.
Clinical example
Question: Which patients discharged after heart failure will be readmitted within 30 days? Design: Prespecify candidate predictors available at discharge, size the study using the Riley criteria for the expected outcome proportion and number of predictor parameters, fit the model, and report discrimination and calibration with bootstrap internal validation.
How to report it
TRIPOD+AI is the current guideline for this design: it updates the 2015 TRIPOD statement, which it replaces, and covers models developed with regression or machine learning methods. Both ask for the source of the data, how predictors were handled and how many events supported how many candidate predictors, and both discrimination and calibration reported for the model as presented rather than for a version tuned on the same data. A reporting guideline is not a design manual: use it to make the study transparent, not as a substitute for careful design. Check the target journal's own requirements as well, which this page cannot know.
- Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes (opens in a new tab) · Stat Med · doi 10.1002/sim.7992
- Minimum sample size for developing a multivariable prediction model: Part I - Continuous outcomes (opens in a new tab) · Stat Med · doi 10.1002/sim.7993
- Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement (opens in a new tab) · Ann Intern Med · doi 10.7326/M14-0697
- TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods (opens in a new tab) · BMJ · doi 10.1136/bmj-2023-078378