Ask Mallard
Analysis method

Cohen’s kappa

4 min read

Cohen’s kappa summarizes agreement between two raters on categories, relative to agreement expected from their separate category frequencies.

Fit

When to use it, and when not to

Use it when

  • Two raters independently classify the same people using the same categories.
  • Ordered categories need weighted kappa, with weights set to reflect the seriousness of disagreement.

Not when

Before you start

What you need

Decide these first
  1. The square cross-tabulation of both ratersUse the same category order for rows and columns. Keep counts of unreadable or missing ratings.
  2. Unweighted or weighted kappaUse unweighted kappa for nominal categories. For ordered categories, state and justify the weights before analysis.
  3. The population and rater rolesSample the cases and raters relevant to the intended use; prevalence affects kappa.
It rests on these
  1. Neither rater sees the other’s answer.How to check: Describe blinding and rate the same material independently.
  2. Each person contributes an independent rating pair.How to check: Account for repeat ratings or clustering rather than treating every rating as a new person.
  3. Missing ratings have a stated explanation.How to check: Report exclusions. Under missing at random (MAR), consider suitable imputation; test sensitivity if missing not at random (MNAR) is plausible.

Plain language

How it works

Find observed agreement from the table’s diagonal. Expected agreement is the sum of products of the two raters’ category proportions. Subtract expected from observed agreement, then divide by one minus expected agreement. Byrt and colleagues show why prevalence and unequal rater tendencies change its interpretation.

Cohen’s kappa in steps
Cross-tabulate independent rating pairs; then Compute observed and marginal expected agreement; then Scale the agreement beyond expectation; then Report kappa, interval and full table.Cohen’s kappaCross-tabulate independent rating pairsCompute observed and marginal expected agreementScale the agreement beyond expectationReport kappa, interval and full tableKappa depends on category frequencies and rater bias. Report the full rating table and raw agreement withkappa and its confidence interval; verbal bands are not clinical thresholds.
Conceptual diagram. It shows the structure of the idea, not data from any study.
Read the result

Weighted kappa is a variant for ordered ratings. It gives partial credit according to a stated weight matrix. Cohen’s original weighted-kappa paper allows scaled disagreement; linear and quadratic weights encode different judgments. Report the chosen weights and an interval from a method suited to the rating design. Agreement does not establish accuracy.

Worked example

A worked example

Invented for illustration
Setting
Two clinicians classify 100 invented scans as positive or negative.
Question
How much do their classifications agree beyond the marginal-frequency benchmark?
Approach
Both say positive on 40 scans and both negative on 40. Each clinician alone says positive on 10 other scans.
Result
They agree on 80 of 100 scans, or 80 in 100. Each rates 50 in 100 positive; expected agreement is 0.50. Kappa is (0.80 minus 0.50) divided by (1 minus 0.50), or 0.60.
What it means
Kappa describes these raters and this case mix. It does not show that either clinician identified the true condition.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

Kappa depends on category frequencies and rater bias. Report the full rating table and raw agreement with kappa and its confidence interval; verbal bands are not clinical thresholds.

  • Treating a low kappa with high raw agreement as a calculation error.Instead: Inspect category frequencies and the table. Rare positive ratings can produce that combination.
  • Choosing weights that make agreement look strongest.Instead: Choose weights for the clinical costs of disagreement, then report them.
  • Replacing the estimate with a label such as substantial.Instead: Report kappa, its confidence interval, raw agreement and the table.

Inside Mallard

How Mallard uses this

Mallard can offer kappa for an agreement question. Plans naming kappa link to this article; its calculator offers unweighted or quadratic-weighted kappa.

No Mallard code example points to this page.
Sample size: Mallard sizes the main comparison with agreement kappa. It does not size adjustment or any secondary analysis.
Calculators: Cohen's κ (agreement) (§7.1).
Atlas node analysis.cohens_kappa, the id a plan cue records.

Work it out

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit (opens in a new tab) · Psychol Bull · doi 10.1037/h0026256
  2. Bias, prevalence and kappa (opens in a new tab) · J Clin Epidemiol · doi 10.1016/0895-4356(93)90018-v
Prepared with AI assistance. Source records were checked against PubMed on 2026-11-06, matching title, authors, year and available DOI. Cohen (1968) supports weighted agreement and partial credit; Byrt, Bishop and Carlin (1993) support the prevalence and bias cautions. Record verification is not independent methodological approval.
Last source review: 2026-11-06
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.