Sample size for estimating kappa
Plan how many independent cases two raters need to estimate binary kappa within a chosen interval half-width.
Fit
When to use it, and when not to
Use it when
- Two raters each give a binary rating to every case.
- You can justify a common positive-rating prevalence and want a confidence interval with a chosen half-width.
Not when
- Ratings are continuous. Plan an intraclass correlation or limits-of-agreement study instead. Read: Intraclass correlation for reliability
- You need weighted kappa, more raters or unequal margins. Use a sizing method that matches that rating design. Read: Cohen’s kappa
Before you start
What you need
- Expected kappaSupply a value strictly between zero and one. Kappa measures agreement beyond the marginal-frequency benchmark.
- Positive-rating prevalenceSupply the expected positive fraction, strictly between zero and one, shared by both raters. This is not disease prevalence.
- Interval half-widthSupply the desired distance from the estimate to either limit, strictly between zero and 0.5.
- Confidence level through alphaAlpha defaults to 0.05, giving 95 percent confidence. Target power is not used.
- Two binary raters have the same expected positive-rating prevalence.How to check: Check prior rating tables for each rater. Different margins need a more general calculation.
- Cases are independent and represent the intended case mix.How to check: Do not count several ratings of one patient as separate independent cases.
- The large-sample variance approximation is adequate.How to check: Rare categories or near-perfect agreement need specialist checks. The built-in minimum does not prove adequacy.
Plain language
How it works
Mallard uses the Bloch–Kraemer binary-rater variance approximation with one shared positive-rating prevalence. Write expected kappa as k and the common positive fraction as p. The variance coefficient is (1−k)×[(1−k)×(1−2k)+k×(2−k)/(2×p×(1−p))]. Multiply it by the squared normal confidence quantile and divide by the squared target half-width.
It rounds upward and imposes a minimum of 20 cases. Both raters assess each case. A narrow interval needs more cases; the category mix also changes the requirement. Choosing a sample-size method distinguishes this precision target from testing a null kappa.
Worked example
A worked example
- Setting
- Two clinicians will each classify every scan as positive or negative.
- Question
- How many independent scans are needed to estimate kappa with a chosen interval half-width?
- Approach
- Enter kappa = 0.6, prev = 0.3, width = 0.1 and alpha = 0.05. This is precision planning.
- Result
- Mallard returns 296 cases, each rated by both clinicians.
- What it means
- The assumed positive-rating fraction is 30 in 100. The target half-width is 0.1 for a 95 percent interval; it is not guaranteed.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
This large-sample precision approximation uses binary ratings and one common positive-rating prevalence. It does not size weighted kappa, multiple raters or unequal rater margins.
- Calling the result 80 percent power.Instead: State the confidence level and target half-width. No power input enters this calculation.
- Confusing positive-rating prevalence with disease prevalence.Instead: Use the raters’ expected positive fraction; no reference-standard truth is required.
- Counting twice because two raters see each case.Instead: Count independent cases. Each case supplies both ratings.
Inside Mallard
How Mallard uses this
Mallard returns total cases for interval precision, with a minimum of 20 cases, and approximate half-width at a supplied total. It does not calculate power.
analysis.kappa_sample_size, the id a plan cue records.Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- 2 x 2 kappa coefficients: measures of agreement or association (opens in a new tab) · Biometrics
- The kappa statistic in reliability studies: use, interpretation, and sample size requirements (opens in a new tab) · Phys Ther