Ask Mallard
Analysis method

Weighted kappa: agreement on an ordered scale

5 min read

Weighted kappa measures agreement between two raters on ordered categories while giving partial credit for disagreements of different sizes.

Fit

When to use it, and when not to

Use it when

  • Two raters independently assign the same ordered categories to each person.
  • A one-category disagreement matters less than a larger disagreement, according to weights chosen in advance.

Not when

Before you start

What you need

Decide these first
  1. A square table in the same category orderRows identify one rater’s categories; columns identify the other’s. Record missing ratings separately.
  2. Prespecified weights and category spacingLinear and quadratic weights use category-step distance. Check whether those steps reflect the clinical cost of disagreement.
  3. Cases, rater roles and interval methodDefine the population and sampling design. Category frequencies affect the estimate and its uncertainty.
It rests on these
  1. Each person contributes an independent rating pair.How to check: Account for repeated ratings or clustered people when estimating uncertainty.
  2. Raters assign scores without seeing each other’s ratings.How to check: Describe blinding and use the same definitions and material.
  3. The ordered categories and weights fit the intended use.How to check: Do not choose weights after seeing which gives the largest kappa.

Plain language

How it works

Let K be the number of ordered categories, and d the category-step distance between ratings. Set f to d divided by (K minus 1). Linear agreement weights are 1 minus f; quadratic weights are 1 minus f squared. Exact matches get full credit; opposite ends get none. Quadratic weights give more credit to intermediate disagreements.

One method, chosen weights

Compute observed weighted agreement from the table. Compute expected weighted agreement from the product of each rater’s category proportions, using the same weights. Weighted kappa is observed minus expected agreement, divided by one minus expected agreement. Cohen introduces scaled disagreement; Li examines the different behavior of linear and quadratic weights. Neither weighting rule proves equal clinical importance of category steps.

The method in steps
Independent ratings in an ordered table; then Choose linear or quadratic agreement weights; then Compute weighted observed and expected agreement; then Report weighted kappa, weights and full table. Use the same weights for observed and expected agreement. Partial credit is a choice within weighted kappa, not a new agreement method.Weighted kappa: agreement on an ordered scaleIndependent ratings in an ordered tableChoose linear or quadratic agreement weightsCompute weighted observed and expected agreementReport weighted kappa, weights and full tableUse the same weights for observed and expected agreement. Partial credit is a choice within weightedkappa, not a new agreement method.
Conceptual diagram. It shows the structure of the idea, not data from any study.

Worked example

A worked example

Invented for illustration
Setting
Two clinicians score 20 independent cases as low, middle or high.
Question
How much weighted agreement exceeds the benchmark from their separate category frequencies?
Approach
Rows follow rater A; columns follow rater B. Low, middle, high counts are [5, 1, 0], [1, 6, 1], [0, 1, 5].
Result
Exact agreement is 16 of 20, or 80 in 100. Linear weighted agreement is 0.900; expected agreement is 0.580; kappa is 0.762. Quadratic values are 0.950, 0.700 and 0.833.
What it means
Quadratic weights give adjacent disagreements more partial credit here. The larger coefficient does not show that the raters changed or that their scores are accurate.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

Choose weights before examining agreement. Linear and quadratic weights define different amounts of partial credit; report the weights, full table, raw agreement and a suitable confidence interval.

  • Treating linear and quadratic kappa as two names for the same answer.Instead: State the weighting rule. Compare coefficients only when their weighting schemes and case mix are clear.
  • Reporting weighted agreement as the fraction of exact matches.Instead: Report exact agreement separately. Partial credit raises weighted agreement without adding exact matches.
  • Using a single-proportion interval for kappa.Instead: Use an interval for weighted kappa that respects the rating design, such as a suitable case bootstrap.

Inside Mallard

How Mallard uses this

Mallard links this article to its kappa calculator, which offers quadratic weights but no linear option. Its displayed interval uses a simplified approximation; confirm uncertainty with a method suited to weighted kappa.

Code example in Mallard: Cohen's and quadratic-weighted kappa for two raters (weighted-kappa-agreement).
Calculators: Cohen's κ (agreement) (§7.1).
Atlas node analysis.weighted_kappa, the id a plan cue records.
Code for this method
Loading…

Work it out

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit (opens in a new tab) · Psychol Bull · doi 10.1037/h0026256
  2. A Note on the Linearly and Quadratically Weighted Kappa Coefficients (opens in a new tab) · Psychometrika · doi 10.1007/s11336-016-9501-5
Prepared with AI assistance. PubMed records checked on 2026-11-07: title, authors, year and DOI matched. Record verified, confirm it supports this use. Cohen introduces weighted kappa and partial credit. Li examines properties of linear and quadratic weighting. No independent professional approval is claimed.
Last source review: 2026-11-07
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.