Comparing two ROC curves
The DeLong test compares two full empirical areas under ROC curves, preserving covariance when both tests score the same patients.
Fit
When to use it, and when not to
Use it when
- You prespecified a comparison of two full areas under receiver operating characteristic curves.
- Your test scores are evaluated against a binary reference diagnosis in paired patients or independent cohorts.
Not when
- You compare sensitivities at one fixed threshold in the same patients. Use McNemar’s test among cases. Read: McNemar’s test
- You need a partial area, smoothed curve or clustered analysis. Use a bootstrap designed for that estimator and sampling unit. Read: The bootstrap
- You only need to describe one test across thresholds. Use an ROC curve. Read: ROC curves and diagnostic thresholds
Before you start
What you need
- Both tests’ scores and reference diagnosesPrespecify which direction suggests disease. Count tied case-noncase scores as half a correct ranking.
- Patient identifiers and sampling structuresamplingStructurePaired tests use the same cases and noncases. Unpaired tests use independent cohorts; partially overlapping cohorts need a different covariance analysis.
- The direction of the area differencesummaryMeasureDefine test A minus test B, the confidence level and whether the hypothesis is two-sided.
- Independent evaluation and a missing-score planAssess locked scores outside training. Keep pairs aligned; justify complete-case analysis or imputation under a stated missingness assumption.
- Different patients are independent.How to check: Repeated visits, lesions or clinic sampling need a method that preserves those clusters.
- Paired scores refer to the same patients in the same order.How to check: Use covariance for paired areas. For independent cohorts, add the two area variances; covariance is zero.
- Enough cases and noncases support large-sample inference.How to check: Tiny samples or areas near perfect separation can give unstable variance estimates. Examine a stratified bootstrap and its limitations.
- Both scores address the same target population.How to check: Differences between unpaired cohorts can reflect case mix, verification or disease severity rather than the tests.
Plain language
How it works
The area under a receiver operating characteristic curve, or AUC, counts case-noncase rankings. DeLong estimates its variance from each case’s and noncase’s contribution. For paired tests, subtract twice their covariance from the sum of variances. Divide the area difference by its standard error for a large-sample normal test.
For unpaired cohorts, the covariance is zero: add both area variances. The example uses normal intervals for both designs. Software may use a Welch approximation for unpaired comparisons; report the implementation. Partial areas, smoothed curves and nested models evaluated on training data need other inference.
Worked example
A worked example
- Setting
- Both tests score the same 6 cases and 6 noncases; higher scores suggest disease. These tiny invented data teach arithmetic, not reliable inference.
- Question
- How does test A’s full area differ from test B’s?
- Approach
- In patient order, case scores are A: 4, 5, 6, 7, 8, 9; B: 3, 4, 5, 6, 7, 8. Noncase scores are A: 1, 2, 3, 5, 6, 7; B: 2, 1, 4, 3, 7, 6. Count ties as half.
- Result
- The areas under the curves are 0.792 for A and 0.722 for B; their difference is 0.069. Paired DeLong standard error is 0.080, z = 0.868, two-sided p = 0.386. The normal 95% difference interval is -0.087 to 0.226.
- What it means
- If B instead came from an independent cohort with these same scores, the unpaired standard error would be 0.207. The unpaired normal 95% interval is -0.337 to 0.476; z = 0.335, p = 0.738. Neither result proves equal accuracy or equal clinical value.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
DeLong compares full empirical AUCs, not clinical utility or sensitivity at a chosen threshold. Preserve paired covariance; independent cohorts need separate variance contributions. Tiny samples, clustered observations and reused training data need other safeguards.
- Comparing separate area intervals to test the difference.Instead: Estimate the interval for the difference itself, with the correct covariance.
- Calling a nonsignificant test proof of equivalence.Instead: Equality testing cannot establish equivalence. Prespecify a clinically justified margin and an appropriate equivalence analysis.
- Treating a larger area as greater clinical benefit.Instead: Check clinically relevant thresholds, calibration and consequences of false results.
Inside Mallard
How Mallard uses this
The method registry links paired diagnostic-test comparison to this article and code. Mallard has no unpaired DeLong calculator.
comparing-two-diagnostic-tests).analysis.delong, the id a plan cue records.How to report it
Report cohort overlap, score direction, evaluation data, both AUCs, their signed difference, its interval and the paired or unpaired test implementation. A reporting guideline is not a design manual: use it to make the study transparent, not as a substitute for careful design. Check the target journal's own requirements as well, which this page cannot know.
Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. (opens in a new tab) · Biometrics
- pROC: an open-source package for R and S+ to analyze and compare ROC curves. (opens in a new tab) · BMC Bioinformatics · doi 10.1186/1471-2105-12-77