Agreement and reliability
What question does this help answer?
Whether a cheaper, faster or less invasive method can replace an established one, and whether a measurement is stable enough to be worth acting on.
Agreement is not correlation, and not accuracy
Correlation asks whether two things move together. Agreement asks whether they give the SAME NUMBER. Accuracy asks whether either is right, which requires a reference standard that an agreement study usually does not have. Confusing the three is the defining error of this literature.
Correlated but not in agreement
Continuous measurements: Bland-Altman
Plot the difference between the two methods against their average, and report the mean difference (the bias) and the limits of agreement, roughly the mean difference plus and minus two standard deviations of the differences. Those limits are the answer: the question is whether a disagreement of that size would change a clinical decision, and that is a clinical judgement made in advance, not a statistical threshold.
Categorical ratings: kappa
- Cohen's kappa: two raters, unordered categories, correcting for agreement expected by chance.
- Weighted kappa: ordered categories, where being one category apart should count less than being three apart.
- Fleiss's kappa: more than two raters.
- Kappa is prevalence dependent, so report the observed agreement and the marginal distributions alongside it.
- The conventional verbal labels for kappa values are arbitrary and were never intended as clinical thresholds.
Repeated continuous measurements: the reliability ICC
The intraclass correlation used here is a reliability coefficient: the proportion of total variance attributable to real differences between subjects. There are ten forms of it, and they answer different questions, so the model, type and definition must all be stated. Note that this is a DIFFERENT statistic from the clustering ICC used to size a cluster-randomized trial, sharing only the acronym.
Why Mallard may recommend it
Mallard may name these methods when the description compares two measurement approaches, assesses observer variation, or asks whether a rating is reproducible.
What to watch out for
Correlation is the wrong tool here and is still the most common one used: two methods that disagree by a constant factor correlate almost perfectly. For categorical ratings, kappa depends on how common each category is, so the same underlying agreement produces very different kappas in different populations, and a high percentage agreement with a low kappa is usually a prevalence artefact rather than a paradox to be explained away.
Clinical example
Question: Can a wrist device replace an arm cuff for blood pressure? Analysis: Bland-Altman with limits of agreement, judged against a clinically acceptable difference stated before the study, not a correlation coefficient between the two devices.
- Statistical methods for assessing agreement between two methods of clinical measurement (opens in a new tab) · Lancet
- A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research (opens in a new tab) · J Chiropr Med · doi 10.1016/j.jcm.2016.02.012
- Interrater reliability: the kappa statistic (opens in a new tab) · Biochem Med (Zagreb)