Ask Mallard
Analysis method

Sample size for estimating kappa

4 min read

Plan how many independent cases two raters need to estimate binary kappa within a chosen interval half-width.

Fit

When to use it, and when not to

Use it when

  • Two raters each give a binary rating to every case.
  • You can justify a common positive-rating prevalence and want a confidence interval with a chosen half-width.

Not when

Before you start

What you need

Decide these first
  1. Expected kappaSupply a value strictly between zero and one. Kappa measures agreement beyond the marginal-frequency benchmark.
  2. Positive-rating prevalenceSupply the expected positive fraction, strictly between zero and one, shared by both raters. This is not disease prevalence.
  3. Interval half-widthSupply the desired distance from the estimate to either limit, strictly between zero and 0.5.
  4. Confidence level through alphaAlpha defaults to 0.05, giving 95 percent confidence. Target power is not used.
It rests on these
  1. Two binary raters have the same expected positive-rating prevalence.How to check: Check prior rating tables for each rater. Different margins need a more general calculation.
  2. Cases are independent and represent the intended case mix.How to check: Do not count several ratings of one patient as separate independent cases.
  3. The large-sample variance approximation is adequate.How to check: Rare categories or near-perfect agreement need specialist checks. The built-in minimum does not prove adequacy.

Plain language

How it works

Mallard uses the Bloch–Kraemer binary-rater variance approximation with one shared positive-rating prevalence. Write expected kappa as k and the common positive fraction as p. The variance coefficient is (1−k)×[(1−k)×(1−2k)+k×(2−k)/(2×p×(1−p))]. Multiply it by the squared normal confidence quantile and divide by the squared target half-width.

From assumptions to a total
Expected kappa and positive-rating prevalence. Then Large-sample interval half-width target. Then Independent cases rated by both raters.Sample size for estimating kappaExpected kappa andpositive-rating prevalenceLarge-sample intervalhalf-width targetIndependent cases rated byboth ratersBinary ratings with a common expected margin; precision, not power.
Conceptual diagram. It shows the structure of the idea, not data from any study.
What the target means

It rounds upward and imposes a minimum of 20 cases. Both raters assess each case. A narrow interval needs more cases; the category mix also changes the requirement. Choosing a sample-size method distinguishes this precision target from testing a null kappa.

Worked example

A worked example

Invented for illustration
Setting
Two clinicians will each classify every scan as positive or negative.
Question
How many independent scans are needed to estimate kappa with a chosen interval half-width?
Approach
Enter kappa = 0.6, prev = 0.3, width = 0.1 and alpha = 0.05. This is precision planning.
Result
Mallard returns 296 cases, each rated by both clinicians.
What it means
The assumed positive-rating fraction is 30 in 100. The target half-width is 0.1 for a 95 percent interval; it is not guaranteed.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

This large-sample precision approximation uses binary ratings and one common positive-rating prevalence. It does not size weighted kappa, multiple raters or unequal rater margins.

  • Calling the result 80 percent power.Instead: State the confidence level and target half-width. No power input enters this calculation.
  • Confusing positive-rating prevalence with disease prevalence.Instead: Use the raters’ expected positive fraction; no reference-standard truth is required.
  • Counting twice because two raters see each case.Instead: Count independent cases. Each case supplies both ratings.

Inside Mallard

How Mallard uses this

Mallard returns total cases for interval precision, with a minimum of 20 cases, and approximate half-width at a supplied total. It does not calculate power.

No Mallard code example points to this page.
Atlas node analysis.kappa_sample_size, the id a plan cue records.
Code for this method
Loading…

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. 2 x 2 kappa coefficients: measures of agreement or association (opens in a new tab) · Biometrics
  2. The kappa statistic in reliability studies: use, interpretation, and sample size requirements (opens in a new tab) · Phys Ther
Prepared with AI assistance. PubMed records were matched by title, authors and year on 2026-11-06: record verified, confirm it supports this use. PubMed returned no DOI for either record. Bloch and Kraemer support binary-kappa sampling theory. Sim and Wright discuss prevalence, rater bias, confidence intervals and sample size. The implemented approximation and minimum were checked against Mallard’s engine. No independent professional approval is claimed.
Last source review: 2026-11-06
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.