Cohen’s kappa
Cohen’s kappa summarizes agreement between two raters on categories, relative to agreement expected from their separate category frequencies.
Fit
When to use it, and when not to
Use it when
- Two raters independently classify the same people using the same categories.
- Ordered categories need weighted kappa, with weights set to reflect the seriousness of disagreement.
Not when
- You have continuous readings. Use Bland-Altman limits or a reliability ICC. Read: Bland-Altman method comparison
- You want accuracy against a reference standard. Use a diagnostic accuracy study. Read: Diagnostic accuracy study
Before you start
What you need
- The square cross-tabulation of both ratersUse the same category order for rows and columns. Keep counts of unreadable or missing ratings.
- Unweighted or weighted kappaUse unweighted kappa for nominal categories. For ordered categories, state and justify the weights before analysis.
- The population and rater rolesSample the cases and raters relevant to the intended use; prevalence affects kappa.
- Neither rater sees the other’s answer.How to check: Describe blinding and rate the same material independently.
- Each person contributes an independent rating pair.How to check: Account for repeat ratings or clustering rather than treating every rating as a new person.
- Missing ratings have a stated explanation.How to check: Report exclusions. Under missing at random (MAR), consider suitable imputation; test sensitivity if missing not at random (MNAR) is plausible.
Plain language
How it works
Find observed agreement from the table’s diagonal. Expected agreement is the sum of products of the two raters’ category proportions. Subtract expected from observed agreement, then divide by one minus expected agreement. Byrt and colleagues show why prevalence and unequal rater tendencies change its interpretation.
Weighted kappa is a variant for ordered ratings. It gives partial credit according to a stated weight matrix. Cohen’s original weighted-kappa paper allows scaled disagreement; linear and quadratic weights encode different judgments. Report the chosen weights and an interval from a method suited to the rating design. Agreement does not establish accuracy.
Worked example
A worked example
- Setting
- Two clinicians classify 100 invented scans as positive or negative.
- Question
- How much do their classifications agree beyond the marginal-frequency benchmark?
- Approach
- Both say positive on 40 scans and both negative on 40. Each clinician alone says positive on 10 other scans.
- Result
- They agree on 80 of 100 scans, or 80 in 100. Each rates 50 in 100 positive; expected agreement is 0.50. Kappa is (0.80 minus 0.50) divided by (1 minus 0.50), or 0.60.
- What it means
- Kappa describes these raters and this case mix. It does not show that either clinician identified the true condition.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
Kappa depends on category frequencies and rater bias. Report the full rating table and raw agreement with kappa and its confidence interval; verbal bands are not clinical thresholds.
- Treating a low kappa with high raw agreement as a calculation error.Instead: Inspect category frequencies and the table. Rare positive ratings can produce that combination.
- Choosing weights that make agreement look strongest.Instead: Choose weights for the clinical costs of disagreement, then report them.
- Replacing the estimate with a label such as substantial.Instead: Report kappa, its confidence interval, raw agreement and the table.
Inside Mallard
How Mallard uses this
Mallard can offer kappa for an agreement question. Plans naming kappa link to this article; its calculator offers unweighted or quadratic-weighted kappa.
analysis.cohens_kappa, the id a plan cue records.Work it out
Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit (opens in a new tab) · Psychol Bull · doi 10.1037/h0026256
- Bias, prevalence and kappa (opens in a new tab) · J Clin Epidemiol · doi 10.1016/0895-4356(93)90018-v