Fleiss’ kappa: agreement among multiple raters
Fleiss’ kappa summarizes agreement among multiple raters who classify the same cases, relative to a benchmark from pooled category frequencies.
Fit
When to use it, and when not to
Use it when
- More than two raters independently classify each case into the same nominal categories.
- Raters are treated as exchangeable, with the same number of ratings per case in the standard calculation.
Not when
- Only two identified raters classify every person. Use Cohen’s kappa with separate rater margins. Read: Cohen’s kappa
- Two raters use ordered categories and disagreements differ in seriousness. Use weighted kappa. Read: Weighted kappa: agreement on an ordered scale
- You need reliability for continuous readings. Choose an intraclass correlation model. Read: Intraclass correlation for reliability
Before you start
What you need
- Category counts within every caseKeep case rows, including unanimous rows. State the number of ratings per case and the missing-rating policy.
- The rater sampling designState whether the same panel rates all cases. Distinct fixed rater roles may require another multirater coefficient.
- The target population and uncertainty methodUse a variance estimate valid beyond the zero-kappa null, or a bootstrap suited to the sampling design.
- Cases are independent sampling units.How to check: Account for repeated cases or clusters. Rater pairs within a case are not independent cases.
- Rater identity is not part of the target contrast.How to check: Check for systematic rater differences before pooling their category frequencies.
- Raters use the same categories and rate independently.How to check: Blind raters to each other. Do not use adjudicated consensus as independent ratings.
Plain language
How it works
Count how many raters place each case in each category. Within a case, divide the number of agreeing rater pairs by all rater pairs. Average across cases. Pool category frequencies across all ratings; expected agreement is the sum of their squares. Fleiss’ kappa is mean agreement minus expected agreement, divided by one minus expected agreement.
The standard calculation gives each case the same number of ratings and treats raters as exchangeable. It ignores category order and rater identity. It is not an average of Cohen’s kappas with separate margins. Gwet and Vanbelle derive uncertainty estimates for multirater kappa beyond the zero-kappa null. A test of zero agreement does not supply a general confidence interval.
Worked example
A worked example
- Setting
- Three raters independently classify six independent cases into A or B.
- Question
- How much does their average pair agreement exceed the pooled-frequency benchmark?
- Approach
- The six case rows give counts in A, B order: [3, 0], [0, 3], [2, 1], [1, 2], [3, 0], [0, 3].
- Result
- Across 18 within-case rater pairs, 14 agree. Mean pair agreement is 0.778. Pooled A and B frequencies are 0.500 each. Expected agreement is 0.500; Fleiss’ kappa is 0.556.
- What it means
- This describes agreement relative to the pooled category mix. It does not identify which rater is correct or establish diagnostic accuracy.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
Fleiss’ kappa uses pooled category frequencies and treats raters as exchangeable. Report the rating design and raw pair agreement; a null-only standard error cannot give a general confidence interval.
- Averaging Cohen’s kappas and calling the answer Fleiss’ kappa.Instead: Use within-case pair agreement and pooled margins. The expected-agreement benchmarks differ.
- Applying the equal-count formula after silently dropping missing ratings.Instead: Report missing ratings and use a method suited to the resulting design.
- Building a confidence interval from a test-only standard error.Instead: Use a general variance method. Gwet and Vanbelle explain why the null-only variance is insufficient.
Inside Mallard
How Mallard uses this
Mallard offers this article through Learn search, related topics and the Atlas. Its two-rater kappa calculator does not calculate Fleiss’ kappa.
analysis.fleiss_kappa, the id a plan cue records.Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Large-Sample Variance of Fleiss Generalized Kappa (opens in a new tab) · Educ Psychol Meas · doi 10.1177/0013164420973080
- Asymptotic variability of (multilevel) multirater kappa coefficients (opens in a new tab) · Stat Methods Med Res · doi 10.1177/0962280218794733