Ask Mallard
Analysis method

Fleiss’ kappa: agreement among multiple raters

5 min read

Fleiss’ kappa summarizes agreement among multiple raters who classify the same cases, relative to a benchmark from pooled category frequencies.

Fit

When to use it, and when not to

Use it when

  • More than two raters independently classify each case into the same nominal categories.
  • Raters are treated as exchangeable, with the same number of ratings per case in the standard calculation.

Not when

Before you start

What you need

Decide these first
  1. Category counts within every caseKeep case rows, including unanimous rows. State the number of ratings per case and the missing-rating policy.
  2. The rater sampling designState whether the same panel rates all cases. Distinct fixed rater roles may require another multirater coefficient.
  3. The target population and uncertainty methodUse a variance estimate valid beyond the zero-kappa null, or a bootstrap suited to the sampling design.
It rests on these
  1. Cases are independent sampling units.How to check: Account for repeated cases or clusters. Rater pairs within a case are not independent cases.
  2. Rater identity is not part of the target contrast.How to check: Check for systematic rater differences before pooling their category frequencies.
  3. Raters use the same categories and rate independently.How to check: Blind raters to each other. Do not use adjudicated consensus as independent ratings.

Plain language

How it works

Count how many raters place each case in each category. Within a case, divide the number of agreeing rater pairs by all rater pairs. Average across cases. Pool category frequencies across all ratings; expected agreement is the sum of their squares. Fleiss’ kappa is mean agreement minus expected agreement, divided by one minus expected agreement.

What the benchmark assumes

The standard calculation gives each case the same number of ratings and treats raters as exchangeable. It ignores category order and rater identity. It is not an average of Cohen’s kappas with separate margins. Gwet and Vanbelle derive uncertainty estimates for multirater kappa beyond the zero-kappa null. A test of zero agreement does not supply a general confidence interval.

The method in steps
Independent category ratings within each case; then Average within-case pair agreement; then Compute expected agreement from pooled frequencies; then Report Fleiss kappa and design-appropriate uncertainty. The benchmark pools category frequencies across raters. Case counts, rater exchangeability and missing ratings must match the stated design.Fleiss’ kappa: agreement among multiple ratersIndependent category ratings within each caseAverage within-case pair agreementCompute expected agreement from pooled frequenciesReport Fleiss kappa and design-appropriate uncertaintyThe benchmark pools category frequencies across raters. Case counts, rater exchangeability and missingratings must match the stated design.
Conceptual diagram. It shows the structure of the idea, not data from any study.

Worked example

A worked example

Invented for illustration
Setting
Three raters independently classify six independent cases into A or B.
Question
How much does their average pair agreement exceed the pooled-frequency benchmark?
Approach
The six case rows give counts in A, B order: [3, 0], [0, 3], [2, 1], [1, 2], [3, 0], [0, 3].
Result
Across 18 within-case rater pairs, 14 agree. Mean pair agreement is 0.778. Pooled A and B frequencies are 0.500 each. Expected agreement is 0.500; Fleiss’ kappa is 0.556.
What it means
This describes agreement relative to the pooled category mix. It does not identify which rater is correct or establish diagnostic accuracy.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

Fleiss’ kappa uses pooled category frequencies and treats raters as exchangeable. Report the rating design and raw pair agreement; a null-only standard error cannot give a general confidence interval.

  • Averaging Cohen’s kappas and calling the answer Fleiss’ kappa.Instead: Use within-case pair agreement and pooled margins. The expected-agreement benchmarks differ.
  • Applying the equal-count formula after silently dropping missing ratings.Instead: Report missing ratings and use a method suited to the resulting design.
  • Building a confidence interval from a test-only standard error.Instead: Use a general variance method. Gwet and Vanbelle explain why the null-only variance is insufficient.

Inside Mallard

How Mallard uses this

Mallard offers this article through Learn search, related topics and the Atlas. Its two-rater kappa calculator does not calculate Fleiss’ kappa.

No Mallard code example points to this page.
Atlas node analysis.fleiss_kappa, the id a plan cue records.

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. Large-Sample Variance of Fleiss Generalized Kappa (opens in a new tab) · Educ Psychol Meas · doi 10.1177/0013164420973080
  2. Asymptotic variability of (multilevel) multirater kappa coefficients (opens in a new tab) · Stat Methods Med Res · doi 10.1177/0962280218794733
Prepared with AI assistance. PubMed records checked on 2026-11-07: title, authors, year and DOI matched. Record verified, confirm it supports this use. Gwet derives a general variance for Fleiss’ kappa. Vanbelle derives multirater uncertainty and discusses coefficient choice. No independent professional approval is claimed.
Last source review: 2026-11-07
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.