Agreement and reliability studies
Why this level?
Specify the research question: intervention, diagnostic accuracy, prognosis, aetiology, or screening intervention. Other question types are outside this hierarchy. NHMRC 2009 sourceAgreement between raters or methods is not one of NHMRC's five research areas at all. There is no exposure and no outcome being compared; there is a measurement being asked how reproducible it is.
What question does this help answer?
It answers whether a measurement can be trusted to be reproducible. That is a precondition for using it, and a separate question from whether it is correct.
When to use it
Use it before relying on a subjective or operator-dependent measurement in a larger study, when introducing a new instrument or scoring system, or when a reference standard is being applied by multiple people.
When not to use it
Do not use it to establish accuracy: for that you need a reference standard and a diagnostic accuracy design. Two raters agreeing perfectly may simply be wrong in the same way.
How it works in plain language
The same subjects are assessed independently more than once and the results compared with a statistic matched to the data type: Cohen's kappa for two raters and a categorical outcome, a weighted kappa when categories are ordered, an intraclass correlation for continuous measurements, and a Bland-Altman plot to show the size and pattern of disagreement across the measurement range. A correlation coefficient is the wrong choice, because two methods can correlate perfectly while one reads consistently higher than the other.
Why Mallard may recommend it
Mallard may recommend an agreement design when the question concerns reproducibility between raters, methods or occasions, rather than the relationship between an exposure and an outcome.
Key assumptions and design requirements
- Ratings are made INDEPENDENTLY; raters who confer produce agreement that measures nothing.
- The statistic matches the data: kappa for categories, weighted kappa when ordered, ICC for continuous.
- The ICC form is stated, since the model, type and definition choices give different values from the same data.
- The subjects span the range and mix of the intended population, because kappa depends on how common the trait is.
- Agreement is reported with a confidence interval, not as a bare point estimate against a verbal label.
What to watch out for
Agreement is not accuracy: two raters can agree perfectly and both be wrong, so a reliability study says nothing about validity. Kappa depends on the prevalence of the trait and can be low despite high raw agreement, and a correlation coefficient is the wrong statistic for agreement because it is unchanged by a systematic offset.
Clinical example
Question: Do two radiologists grade the same scans the same way? Design: Both read a set of scans independently, blinded to each other and to outcome, and agreement is reported as a weighted kappa with a confidence interval because the grades are ordered.
How to report it
GRRAS is the guideline for this design. It sets out fifteen issues under the headings a paper already uses, written because reliability and agreement reports so often left out how the sample was selected, how the study was designed and how the analysis was done. Give the agreement statistic with a confidence interval rather than as a bare point estimate. A reporting guideline is not a design manual: use it to make the study transparent, not as a substitute for careful design. Check the target journal's own requirements as well, which this page cannot know.
- Interrater reliability: the kappa statistic (opens in a new tab) · Biochem Med (Zagreb)
- A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research (opens in a new tab) · J Chiropr Med · doi 10.1016/j.jcm.2016.02.012
- Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed (opens in a new tab) · J Clin Epidemiol · doi 10.1016/j.jclinepi.2010.03.002