Intraclass correlation for reliability
A reliability intraclass correlation compares variation between patients with total variation under a specified model for repeated measurements.
Fit
When to use it, and when not to
Use it when
- You have repeated continuous ratings of the same patients.
- You want reliability for a single rating or an average of a stated set of ratings.
Not when
- Your ICC describes similarity of patients within clinics for study sizing. Use the clustering ICC. Read: Clustering and the ICC
- You need differences in clinical units. Use Bland-Altman limits of agreement. Read: Bland-Altman method comparison
- You have nominal categories. Use kappa for two raters. Read: Bland-Altman method comparison
Before you start
What you need
- Who rates which patientsState whether the same raters assess all patients and whether inference extends to other raters.
- Absolute agreement or consistencyAbsolute agreement counts systematic rater offsets as disagreement. Consistency can disregard those offsets.
- One rating or an averageName the number of ratings in the average. Reliability of an average is not reliability of one reading.
- The variance model matches the rating design.How to check: Check the assignment of raters, residual plots and whether variability changes with measurement size.
- Patients span the intended clinical range.How to check: Describe case mix. Between-patient spread changes the ICC even when measurement error stays the same.
- Missing ratings have a stated assumption.How to check: Under missing at random (MAR), consider a fitted model using incomplete records; examine sensitivity if missing not at random (MNAR) is plausible.
Plain language
How it works
ICC here is the reliability statistic, not the clustering ICC used to size a cluster trial. Shrout and Fleiss define distinct forms for different rating designs. Choose the model first: who rates whom, which raters you generalize to, and whether the clinical decision uses one reading or an average.
For a crossed random-rater model, absolute agreement includes rater variance in the denominator; consistency excludes systematic rater offsets. These answer different questions. Rousson explains how interpretations also differ when the raters are fixed. Report the full form and interval, not a bare acronym or a universal quality label.
Worked example
A worked example
- Setting
- A model for invented continuous ratings has variance components of 36 for patients, 4 for raters and 5 for residual error.
- Question
- How reliable is one rating from a rater drawn from the intended rater population?
- Approach
- Use a two-way random-effects model with absolute agreement for a single rating. Add patient, rater and error variances for the denominator.
- Result
- Total variance is 45. Patient differences account for 36 of 45 variance units, or 80 in 100. The reliability ICC is 0.80.
- What it means
- Rater and residual variation account for the remaining 9 variance units. Report the model and confidence interval; the coefficient alone cannot establish clinical acceptability.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
An ICC without its model, agreement definition and rating unit is ambiguous. A high reliability ICC can coexist with clinically important differences in measurement units.
- Writing only ICC in the results.Instead: State the model, absolute agreement or consistency, and single or average rating, with a confidence interval.
- Using an average-rating ICC to justify one reading.Instead: Report reliability for the unit actually used in practice.
- Treating a high ICC as proof that readings are close in clinical units.Instead: Report error or agreement in measurement units as well.
Inside Mallard
How Mallard uses this
Mallard’s method list may include a reliability ICC for an agreement question with continuous measurements. The ICC method links to this article.
intraclass-correlation).analysis.reliability_icc, the id a plan cue records.Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Intraclass correlations: uses in assessing rater reliability (opens in a new tab) · Psychol Bull · doi 10.1037//0033-2909.86.2.420
- Assessing inter-rater reliability when the raters are fixed: Two concepts and two estimates (opens in a new tab) · Biom J · doi 10.1002/bimj.201000066