Ask Mallard
Analysis method

Intraclass correlation for reliability

4 min read

A reliability intraclass correlation compares variation between patients with total variation under a specified model for repeated measurements.

Fit

When to use it, and when not to

Use it when

  • You have repeated continuous ratings of the same patients.
  • You want reliability for a single rating or an average of a stated set of ratings.

Not when

Before you start

What you need

Decide these first
  1. Who rates which patientsState whether the same raters assess all patients and whether inference extends to other raters.
  2. Absolute agreement or consistencyAbsolute agreement counts systematic rater offsets as disagreement. Consistency can disregard those offsets.
  3. One rating or an averageName the number of ratings in the average. Reliability of an average is not reliability of one reading.
It rests on these
  1. The variance model matches the rating design.How to check: Check the assignment of raters, residual plots and whether variability changes with measurement size.
  2. Patients span the intended clinical range.How to check: Describe case mix. Between-patient spread changes the ICC even when measurement error stays the same.
  3. Missing ratings have a stated assumption.How to check: Under missing at random (MAR), consider a fitted model using incomplete records; examine sensitivity if missing not at random (MNAR) is plausible.

Plain language

How it works

ICC here is the reliability statistic, not the clustering ICC used to size a cluster trial. Shrout and Fleiss define distinct forms for different rating designs. Choose the model first: who rates whom, which raters you generalize to, and whether the clinical decision uses one reading or an average.

Intraclass correlation for reliability in steps
Specify targets, raters and intended use; then Choose model, definition and rating unit; then Estimate variance components and interval; then Interpret reliability for that target range.Intraclass correlation for reliabilitySpecify targets, raters and intended useChoose model, definition and rating unitEstimate variance components and intervalInterpret reliability for that target rangeAn ICC without its model, agreement definition and rating unit is ambiguous. A high reliability ICC cancoexist with important differences in measurement units.
Conceptual diagram. It shows the structure of the idea, not data from any study.
Read the result

For a crossed random-rater model, absolute agreement includes rater variance in the denominator; consistency excludes systematic rater offsets. These answer different questions. Rousson explains how interpretations also differ when the raters are fixed. Report the full form and interval, not a bare acronym or a universal quality label.

Worked example

A worked example

Invented for illustration
Setting
A model for invented continuous ratings has variance components of 36 for patients, 4 for raters and 5 for residual error.
Question
How reliable is one rating from a rater drawn from the intended rater population?
Approach
Use a two-way random-effects model with absolute agreement for a single rating. Add patient, rater and error variances for the denominator.
Result
Total variance is 45. Patient differences account for 36 of 45 variance units, or 80 in 100. The reliability ICC is 0.80.
What it means
Rater and residual variation account for the remaining 9 variance units. Report the model and confidence interval; the coefficient alone cannot establish clinical acceptability.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

An ICC without its model, agreement definition and rating unit is ambiguous. A high reliability ICC can coexist with clinically important differences in measurement units.

  • Writing only ICC in the results.Instead: State the model, absolute agreement or consistency, and single or average rating, with a confidence interval.
  • Using an average-rating ICC to justify one reading.Instead: Report reliability for the unit actually used in practice.
  • Treating a high ICC as proof that readings are close in clinical units.Instead: Report error or agreement in measurement units as well.

Inside Mallard

How Mallard uses this

Mallard’s method list may include a reliability ICC for an agreement question with continuous measurements. The ICC method links to this article.

Code example in Mallard: Intraclass correlation for inter-rater reliability, ICC(A,1) (intraclass-correlation).
Atlas node analysis.reliability_icc, the id a plan cue records.
Code for this method
Loading…

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. Intraclass correlations: uses in assessing rater reliability (opens in a new tab) · Psychol Bull · doi 10.1037//0033-2909.86.2.420
  2. Assessing inter-rater reliability when the raters are fixed: Two concepts and two estimates (opens in a new tab) · Biom J · doi 10.1002/bimj.201000066
Prepared with AI assistance. Source records were checked against PubMed on 2026-11-06, matching title, authors, year and available DOI. Shrout and Fleiss (1979) support design-specific ICC forms and intervals; Rousson (2011) supports the fixed-rater interpretation and agreement-versus-consistency distinction. Record verification is not independent methodological approval.
Last source review: 2026-11-06
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.