Weighted kappa: agreement on an ordered scale
Weighted kappa measures agreement between two raters on ordered categories while giving partial credit for disagreements of different sizes.
Fit
When to use it, and when not to
Use it when
- Two raters independently assign the same ordered categories to each person.
- A one-category disagreement matters less than a larger disagreement, according to weights chosen in advance.
Not when
- Categories have no meaningful order. Use unweighted Cohen’s kappa. Read: Cohen’s kappa
- More than two raters classify each person into nominal categories. Consider Fleiss’ kappa. Read: Fleiss’ kappa: agreement among multiple raters
- Readings are continuous and you want reliability. Choose the appropriate intraclass correlation. Read: Intraclass correlation for reliability
Before you start
What you need
- A square table in the same category orderRows identify one rater’s categories; columns identify the other’s. Record missing ratings separately.
- Prespecified weights and category spacingLinear and quadratic weights use category-step distance. Check whether those steps reflect the clinical cost of disagreement.
- Cases, rater roles and interval methodDefine the population and sampling design. Category frequencies affect the estimate and its uncertainty.
- Each person contributes an independent rating pair.How to check: Account for repeated ratings or clustered people when estimating uncertainty.
- Raters assign scores without seeing each other’s ratings.How to check: Describe blinding and use the same definitions and material.
- The ordered categories and weights fit the intended use.How to check: Do not choose weights after seeing which gives the largest kappa.
Plain language
How it works
Let K be the number of ordered categories, and d the category-step distance between ratings. Set f to d divided by (K minus 1). Linear agreement weights are 1 minus f; quadratic weights are 1 minus f squared. Exact matches get full credit; opposite ends get none. Quadratic weights give more credit to intermediate disagreements.
Compute observed weighted agreement from the table. Compute expected weighted agreement from the product of each rater’s category proportions, using the same weights. Weighted kappa is observed minus expected agreement, divided by one minus expected agreement. Cohen introduces scaled disagreement; Li examines the different behavior of linear and quadratic weights. Neither weighting rule proves equal clinical importance of category steps.
Worked example
A worked example
- Setting
- Two clinicians score 20 independent cases as low, middle or high.
- Question
- How much weighted agreement exceeds the benchmark from their separate category frequencies?
- Approach
- Rows follow rater A; columns follow rater B. Low, middle, high counts are [5, 1, 0], [1, 6, 1], [0, 1, 5].
- Result
- Exact agreement is 16 of 20, or 80 in 100. Linear weighted agreement is 0.900; expected agreement is 0.580; kappa is 0.762. Quadratic values are 0.950, 0.700 and 0.833.
- What it means
- Quadratic weights give adjacent disagreements more partial credit here. The larger coefficient does not show that the raters changed or that their scores are accurate.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
Choose weights before examining agreement. Linear and quadratic weights define different amounts of partial credit; report the weights, full table, raw agreement and a suitable confidence interval.
- Treating linear and quadratic kappa as two names for the same answer.Instead: State the weighting rule. Compare coefficients only when their weighting schemes and case mix are clear.
- Reporting weighted agreement as the fraction of exact matches.Instead: Report exact agreement separately. Partial credit raises weighted agreement without adding exact matches.
- Using a single-proportion interval for kappa.Instead: Use an interval for weighted kappa that respects the rating design, such as a suitable case bootstrap.
Inside Mallard
How Mallard uses this
Mallard links this article to its kappa calculator, which offers quadratic weights but no linear option. Its displayed interval uses a simplified approximation; confirm uncertainty with a method suited to weighted kappa.
weighted-kappa-agreement).analysis.weighted_kappa, the id a plan cue records.Work it out
Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit (opens in a new tab) · Psychol Bull · doi 10.1037/h0026256
- A Note on the Linearly and Quadratically Weighted Kappa Coefficients (opens in a new tab) · Psychometrika · doi 10.1007/s11336-016-9501-5