Choosing an agreement statistic
Use limits of agreement for differences in units, kappa for category agreement, and a specified intraclass correlation for reliability.
Fit
When it matters, and when it does not
It matters when
- You compare raters, instruments or repeated measurements on the same targets.
- You need to decide whether disagreements are small enough for the intended use.
It matters less when
- You only ask whether different variables move together. Consider a correlation coefficient. Read: Choosing a correlation coefficient
- One test is judged against a reference diagnosis. Use diagnostic accuracy measures. Read: Diagnostic accuracy study
Before you start
What to check in your study
- The measurement scaleUse category agreement for labels and differences in units for continuous readings.
- Which raters and which readingSpecify fixed or sampled raters, and a single reading or an average.
- What disagreement is acceptableSet clinical limits before inspecting results.
- The limits model describes the measurement differences.How to check: Inspect bias and spread across the measurement range. Repeated measurements per person require an extension.
- The intraclass correlation model matches the design.How to check: Name the model, single or average reading, and absolute agreement or consistency.
Plain language
How it works
Bland-Altman analysis shows average bias and limits that describe the spread of individual paired differences. Under its simple normal-differences model, limits are the mean difference plus or minus about two standard deviations. Compare them with clinical tolerances.
Cohen kappa adjusts agreement between two raters for agreement expected from their marginal category counts. Weighted kappa gives partial credit to specified disagreements, often for ordered categories. Report the weights.
An intraclass correlation coefficient, or ICC, compares between-target variation with total variation under a named model. Its value depends on the sample range. Reliability for ranking targets and absolute agreement for replacing a device are different questions.
Worked example
A worked example
- Setting
- Two devices measure pressure on the same samples.
- Question
- Can a high reliability score justify swapping devices?
- Approach
- Device A reads 10, 20 and 30 units. Device B reads 15, 25 and 35 units.
- Result
- The mean rises from 20 to 25 units, a difference of 5 units. All paired differences equal 5.
- What it means
- Correlation is 1, yet B is always 5 units higher. With no spread in these invented differences, the estimated agreement limits both equal 5.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
A high correlation or reliability coefficient does not show that measurement differences are small enough for clinical use.
- Reporting an intraclass correlation without its definition.Instead: An absolute-agreement ICC penalizes systematic differences. A consistency ICC can ignore an additive rater shift.
- Reading kappa without the category counts.Instead: Show the cross-table and raw agreement. Category prevalence and marginal totals affect kappa.
Inside Mallard
How Mallard uses this
Mallard lists Choosing an agreement statistic under Core research concepts.
concept.choosing_agreement, the id a plan cue records.Work it out
Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Statistical methods for assessing agreement between two methods of clinical measurement (opens in a new tab) · Lancet
- Intraclass correlations: uses in assessing rater reliability (opens in a new tab) · Psychol Bull · doi 10.1037//0033-2909.86.2.420
- Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit (opens in a new tab) · Psychol Bull · doi 10.1037/h0026256