Ask Mallard
Core concept

Choosing an agreement statistic

5 min read

Use limits of agreement for differences in units, kappa for category agreement, and a specified intraclass correlation for reliability.

Fit

When it matters, and when it does not

It matters when

  • You compare raters, instruments or repeated measurements on the same targets.
  • You need to decide whether disagreements are small enough for the intended use.

It matters less when

Before you start

What to check in your study

Decide these first
  1. The measurement scaleUse category agreement for labels and differences in units for continuous readings.
  2. Which raters and which readingSpecify fixed or sampled raters, and a single reading or an average.
  3. What disagreement is acceptableSet clinical limits before inspecting results.
It rests on these
  1. The limits model describes the measurement differences.How to check: Inspect bias and spread across the measurement range. Repeated measurements per person require an extension.
  2. The intraclass correlation model matches the design.How to check: Name the model, single or average reading, and absolute agreement or consistency.

Plain language

How it works

Bland-Altman analysis shows average bias and limits that describe the spread of individual paired differences. Under its simple normal-differences model, limits are the mean difference plus or minus about two standard deviations. Compare them with clinical tolerances.

The choice at a glance
Differences in units: Clinical tolerance for paired differences leads to Bland-Altman limits of agreement; Category agreement: Labels, margins and chosen weights leads to Cohen kappa or weighted kappa; Reliability: Raters, reading unit and agreement definition leads to Specified reliability ICC.Differences in unitsClinical tolerance for paired differencesBland-Altman limits of agreementUse the study question and assumptions together.Category agreementLabels, margins and chosen weightsCohen kappa or weighted kappaUse the study question and assumptions together.ReliabilityRaters, reading unit and agreement definitionSpecified reliability ICCUse the study question and assumptions together.
Conceptual diagram. It shows the structure of the idea, not data from any study.
Reading the choice

Cohen kappa adjusts agreement between two raters for agreement expected from their marginal category counts. Weighted kappa gives partial credit to specified disagreements, often for ordered categories. Report the weights.

What decides it

An intraclass correlation coefficient, or ICC, compares between-target variation with total variation under a named model. Its value depends on the sample range. Reliability for ranking targets and absolute agreement for replacing a device are different questions.

Worked example

A worked example

Invented for illustration
Setting
Two devices measure pressure on the same samples.
Question
Can a high reliability score justify swapping devices?
Approach
Device A reads 10, 20 and 30 units. Device B reads 15, 25 and 35 units.
Result
The mean rises from 20 to 25 units, a difference of 5 units. All paired differences equal 5.
What it means
Correlation is 1, yet B is always 5 units higher. With no spread in these invented differences, the estimated agreement limits both equal 5.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

A high correlation or reliability coefficient does not show that measurement differences are small enough for clinical use.

  • Reporting an intraclass correlation without its definition.Instead: An absolute-agreement ICC penalizes systematic differences. A consistency ICC can ignore an additive rater shift.
  • Reading kappa without the category counts.Instead: Show the cross-table and raw agreement. Category prevalence and marginal totals affect kappa.

Inside Mallard

How Mallard uses this

Mallard lists Choosing an agreement statistic under Core research concepts.

No Mallard code example points to this page.
Calculators: Cohen's κ (agreement) (§7.1), Bland–Altman (method comparison) (§7.2).
Atlas node concept.choosing_agreement, the id a plan cue records.

Work it out

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. Statistical methods for assessing agreement between two methods of clinical measurement (opens in a new tab) · Lancet
  2. Intraclass correlations: uses in assessing rater reliability (opens in a new tab) · Psychol Bull · doi 10.1037//0033-2909.86.2.420
  3. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit (opens in a new tab) · Psychol Bull · doi 10.1037/h0026256
Prepared with AI assistance. Source titles, authors, publication years and available DOIs were verified against PubMed on 2026-11-06. No independent professional approval is claimed. Bland and Altman develop differences and agreement limits. Shrout and Fleiss distinguish reliability ICC forms. Cohen develops weighted kappa. PubMed lists no DOI for the Bland-Altman record.
Last source review: 2026-11-06
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.