Ask Mallard
Analysis method

Comparing two ROC curves

5 min read

The DeLong test compares two full empirical areas under ROC curves, preserving covariance when both tests score the same patients.

Fit

When to use it, and when not to

Use it when

  • You prespecified a comparison of two full areas under receiver operating characteristic curves.
  • Your test scores are evaluated against a binary reference diagnosis in paired patients or independent cohorts.

Not when

Before you start

What you need

Decide these first
  1. Both tests’ scores and reference diagnosesPrespecify which direction suggests disease. Count tied case-noncase scores as half a correct ranking.
  2. Patient identifiers and sampling structuresamplingStructurePaired tests use the same cases and noncases. Unpaired tests use independent cohorts; partially overlapping cohorts need a different covariance analysis.
  3. The direction of the area differencesummaryMeasureDefine test A minus test B, the confidence level and whether the hypothesis is two-sided.
  4. Independent evaluation and a missing-score planAssess locked scores outside training. Keep pairs aligned; justify complete-case analysis or imputation under a stated missingness assumption.
It rests on these
  1. Different patients are independent.How to check: Repeated visits, lesions or clinic sampling need a method that preserves those clusters.
  2. Paired scores refer to the same patients in the same order.How to check: Use covariance for paired areas. For independent cohorts, add the two area variances; covariance is zero.
  3. Enough cases and noncases support large-sample inference.How to check: Tiny samples or areas near perfect separation can give unstable variance estimates. Examine a stratified bootstrap and its limitations.
  4. Both scores address the same target population.How to check: Differences between unpaired cohorts can reflect case mix, verification or disease severity rather than the tests.

Plain language

How it works

The area under a receiver operating characteristic curve, or AUC, counts case-noncase rankings. DeLong estimates its variance from each case’s and noncase’s contribution. For paired tests, subtract twice their covariance from the sum of variances. Divide the area difference by its standard error for a large-sample normal test.

Independent cohorts and limits

For unpaired cohorts, the covariance is zero: add both area variances. The example uses normal intervals for both designs. Software may use a Welch approximation for unpaired comparisons; report the implementation. Partial areas, smoothed curves and nested models evaluated on training data need other inference.

A separate comparison step
Full empirical areas. Then preserve paired covariance or add independent variances. Then estimate an area difference and its uncertainty.Comparing two ROC areasFull empirical areasPaired covariance orindependent variancesArea difference andintervalThe sampling structure determines the variance of the difference. Full area comparisons do not establishequal performance at one threshold or equal clinical value.
Conceptual diagram. It shows the structure of the idea, not data from any study.
The curve being summarized
1 − specificitySensitivityAUC = 0.80
Illustrative placeholder. Every value in this figure is invented to show the shape of the result, and is not an estimate of anything.

Worked example

A worked example

Invented for illustration
Setting
Both tests score the same 6 cases and 6 noncases; higher scores suggest disease. These tiny invented data teach arithmetic, not reliable inference.
Question
How does test A’s full area differ from test B’s?
Approach
In patient order, case scores are A: 4, 5, 6, 7, 8, 9; B: 3, 4, 5, 6, 7, 8. Noncase scores are A: 1, 2, 3, 5, 6, 7; B: 2, 1, 4, 3, 7, 6. Count ties as half.
Result
The areas under the curves are 0.792 for A and 0.722 for B; their difference is 0.069. Paired DeLong standard error is 0.080, z = 0.868, two-sided p = 0.386. The normal 95% difference interval is -0.087 to 0.226.
What it means
If B instead came from an independent cohort with these same scores, the unpaired standard error would be 0.207. The unpaired normal 95% interval is -0.337 to 0.476; z = 0.335, p = 0.738. Neither result proves equal accuracy or equal clinical value.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

DeLong compares full empirical AUCs, not clinical utility or sensitivity at a chosen threshold. Preserve paired covariance; independent cohorts need separate variance contributions. Tiny samples, clustered observations and reused training data need other safeguards.

  • Comparing separate area intervals to test the difference.Instead: Estimate the interval for the difference itself, with the correct covariance.
  • Calling a nonsignificant test proof of equivalence.Instead: Equality testing cannot establish equivalence. Prespecify a clinically justified margin and an appropriate equivalence analysis.
  • Treating a larger area as greater clinical benefit.Instead: Check clinically relevant thresholds, calibration and consequences of false results.

Inside Mallard

How Mallard uses this

The method registry links paired diagnostic-test comparison to this article and code. Mallard has no unpaired DeLong calculator.

Code example in Mallard: Comparing two diagnostic tests on the same patients (comparing-two-diagnostic-tests).
Atlas node analysis.delong, the id a plan cue records.

How to report it

Report cohort overlap, score direction, evaluation data, both AUCs, their signed difference, its interval and the paired or unpaired test implementation. A reporting guideline is not a design manual: use it to make the study transparent, not as a substitute for careful design. Check the target journal's own requirements as well, which this page cannot know.

Code for this method
Loading…

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. (opens in a new tab) · Biometrics
  2. pROC: an open-source package for R and S+ to analyze and compare ROC curves. (opens in a new tab) · BMC Bioinformatics · doi 10.1186/1471-2105-12-77
Prepared with AI assistance. PubMed records checked by title, authors, year and available DOI on 2026-11-07: record verified, confirm it supports this use. No independent professional approval is claimed. The classic DeLong paper is indexed in PubMed. Robin describes paired and unpaired ROC inference in pROC. The worked unpaired calculation uses the stated large-sample normal approximation.
Last source review: 2026-11-07
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.