ROC curves and diagnostic thresholds
What question does this help answer?
How well a measurement distinguishes people who have a condition from people who do not, without yet committing to a decision threshold.
What the area means
The area under the curve equals the probability that a randomly chosen person WITH the condition has a higher test value than a randomly chosen person without it. That is a genuinely useful quantity and a limited one: 0.5 is coin-flipping, 1.0 is perfect separation, and the number says nothing about whether the test's absolute values can be believed.
What an ROC curve looks like
Discrimination is not calibration
Discrimination is about order; calibration is about whether a predicted 20 percent risk corresponds to 20 percent of such people having the outcome. A model can rank perfectly and still be badly miscalibrated, and it is calibration that a clinician relies on when quoting a risk to a patient. Report both.
Choosing a threshold is a clinical decision
The Youden index treats a false positive and a false negative as equally bad, which they almost never are. Missing a treatable cancer and over-investigating a well patient carry very different costs, and the threshold should follow from those costs. Decision curve analysis makes this explicit by evaluating net benefit across the range of thresholds a clinician might plausibly hold.
Predictive values depend on prevalence
- Sensitivity and specificity are properties of the test in the studied population and are relatively stable across settings.
- Positive and negative predictive value depend on how common the condition is, so they change between a specialist clinic and primary care with the same test.
- Likelihood ratios combine the two and let a clinician move from pre-test to post-test probability directly.
- Reporting predictive values without the prevalence they were computed at leaves them uninterpretable elsewhere.
Why Mallard may recommend it
Mallard may name ROC analysis when the question is how well a continuous marker or score identifies a condition, and it will insist on a prespecified threshold when the study is about a decision rather than about ranking.
Intervals for the area, and comparing two tests
The confidence interval for an area under the curve is usually calculated by DeLong's method, which makes no assumption about the distribution of the marker. When two tests are measured on the SAME patients, their areas are correlated, because both are judged against the same people, and comparing them as if they came from separate samples usually overstates the uncertainty of the difference. DeLong's method for correlated curves accounts for this and is the usual test for a paired comparison of areas. A paired comparison of sensitivity at a prespecified threshold uses McNemar's test among the patients with the condition, and of specificity among those without. A difference in areas can come from parts of the curve no one would use clinically, so compare sensitivity and specificity at the threshold that matters as well. With few patients who have the condition, or an area close to 1, the usual interval can be unreliable, and a bootstrap interval is a reasonable check.
What to watch out for
The area under the ROC curve measures RANKING only: it is unchanged if every predicted probability is doubled, so a model can have an excellent AUC and be systematically wrong about absolute risk. It also ignores what the two kinds of error cost, which is the whole substance of choosing a threshold, and a cutoff chosen to maximise the Youden index in the same data that produced the curve is optimistically biased and needs validating elsewhere.
Clinical example
Question: How well does a triage score identify sepsis in the emergency department? Analysis: Report the ROC curve with the area and its confidence interval, sensitivity and specificity at a threshold fixed in advance, and a calibration plot, since a score used to escalate care must be right about absolute risk.
- Receiver operating characteristic curve in diagnostic test assessment (opens in a new tab) · J Thorac Oncol · doi 10.1097/JTO.0b013e3181ec173d
- Receiver Operating Characteristic (ROC) Curve Analysis for Medical Diagnostic Test Evaluation (opens in a new tab) · Caspian J Intern Med
- Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach (opens in a new tab) · Biometrics