Ask Mallard
Core concept

Measurement properties and the designs that test them

4 min read

A measurement validation study shows what a score measures, in which people and for which use, and how consistently and accurately it measures it.

Fit

When it matters, and when it does not

It matters when

  • You need a questionnaire or rating scale, and none has evidence for your patients, language and use.
  • You are translating, shortening or adapting an existing instrument for a new group.
  • You plan to compare scores between groups or over time, and need evidence that such comparisons are fair.

It matters less when

Before you start

What to check in your study

Decide these first
  1. The construct, the people and the useSay what the score should capture, in whom, and which decision it will inform. Validity belongs to that use, not to the instrument.
  2. How content validity will be shownAsk patients and experts whether the items are relevant, complete and understood, using cognitive interviews, before any statistics.
  3. Which measurement properties you will studySuch as structure, internal consistency, test-retest reliability, measurement error, construct validity, responsiveness and measurement invariance. Each needs its own design.
  4. Hypotheses fixed in advanceFor construct validity and responsiveness, state the expected correlations or group differences, and their direction, before the data are in.
  5. A missing-data plan that starts from an assumptionDecide why items or whole questionnaires may be missing before choosing a method. The scoring rules say what to do with a few blank items.
It rests on these
  1. For test-retest reliability, the construct does not change between the two occasions.How to check: Choose an interval short enough for stability and long enough that people do not recall their first answers. Ask whether anything changed.
  2. Responsiveness is judged in people whose construct really changed.How to check: Use an anchor, such as a patient's own rating of change, and compare the score's change with what you predicted for each group.
  3. A score means the same thing in each group you will compare.How to check: Test measurement invariance across languages, sexes or ages before comparing their scores. Items can behave differently in different groups.
  4. Missing items and questionnaires are handled under a stated assumption.How to check: People who are tired, unwell or struggle with reading leave more items blank, so missing completely at random (MCAR) is rarely plausible. Missing at random (MAR) means the gaps depend only on recorded factors, such as other items and age. Under MAR, multiple imputation can help, but it is not automatically better than the instrument's own scoring rule. If people skip an item because their answer would be extreme, missing not at random (MNAR) is plausible. Then add a tipping-point analysis, which shifts the imputed answers until the conclusion changes.

Plain language

How it works

Start with the construct and the use the score will serve. Build or choose items with patients and experts, and check that people understand them. Then study each measurement property with its own design: structure, internal consistency, reliability over repeated measurement, measurement error, and validity against hypotheses set in advance.

Responsiveness and invariance

Responsiveness is whether the score changes when the construct changes. Test it in people expected to change, against predictions made in advance. Measurement invariance is whether a score means the same thing in different groups, such as two languages. Without it, a difference between groups may come from the questionnaire rather than the people.

Why a high correlation is not enough
Measurements from method A against method B lie along a tight straight line, but above the dashed line of perfect agreement by a constant amount: highly correlated, not in agreement.Correlation is not agreement: a constant offset between two methodsMethod AMethod BPerfect agreementTightly correlated, yet method B always reads higherTwo methods can correlate almost perfectly and never agree. Bland-Altman limits of agreement show theoffset directly.
Conceptual diagram. It shows the structure of the idea, not data from any study.

Worked example

A worked example

Invented for illustration
Setting
A fatigue questionnaire for adults with cancer is tested in 150 patients starting chemotherapy.
Question
Does the score change by more than its measurement error when patients say their fatigue got worse?
Approach
A test-retest study gave a smallest detectable change of 6 points. The team predicted, in advance, that patients reporting worse fatigue at six weeks would change by more than that.
Result
Among the 48 patients who reported worse fatigue, the mean score rose from 54 to 62, a change of 8 points.
What it means
The change of 8 points is larger than the 6 points the scale could detect, as predicted. That is one piece of evidence for responsiveness, not proof that the scale is valid.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

A high Cronbach alpha or correlation with another score does not establish validity. Evidence must support the intended construct, population, language and use.

  • Treating a high Cronbach's alpha as proof that a scale is valid.Instead: Alpha rises with the number of items and does not show that they measure one thing. Validity needs content work and hypotheses about the score.
  • Using a translated or shortened version as if it carried the original's evidence.Instead: Check content and measurement invariance for the new version, in the people who will use it.
  • Dropping respondents with any missing item responses.Instead: Those left may be healthier or better readers, which is selection bias. Apply the scoring rule for blank items, or use multiple imputation.

Inside Mallard

How Mallard uses this

Plans do not link to this concept yet. The kappa and Bland-Altman calculators sit beside this concept. When a plan names factor analysis or item response theory, Mallard's audit rates its complexity very high.

No Mallard code example points to this page.
Calculators: Cohen's κ (agreement) (§7.1), Bland–Altman (method comparison) (§7.2).
Atlas node concept.measurement_properties, the id a plan cue records.

How to report it

COSMIN reporting guideline, in its 2025 version 2.0, covers studies of the measurement properties of patient-reported outcome measures, and GRRAS covers reliability and agreement studies of any scale, instrument or classification. GRRAS was written because such reports so often left out how the sample was selected, how the study was designed and how the analysis was done, so state all three for every property you assess. A reporting guideline is not a design manual: use it to make the study transparent, not as a substitute for careful design. Check the target journal's own requirements as well, which this page cannot know.

Work it out

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study (opens in a new tab) · Qual Life Res · doi 10.1007/s11136-018-1829-0
  2. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study (opens in a new tab) · Qual Life Res · doi 10.1007/s11136-010-9606-8
  3. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes (opens in a new tab) · J Clin Epidemiol · doi 10.1016/j.jclinepi.2010.02.006
  4. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls (opens in a new tab) · BMJ · doi 10.1136/bmj.b2393
  5. COSMIN reporting guideline for studies on measurement properties of patient-reported outcome measures: version 2.0 (opens in a new tab) · Qual Life Res · doi 10.1007/s11136-025-03950-x
  6. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed (opens in a new tab) · J Clin Epidemiol · doi 10.1016/j.jclinepi.2010.03.002
Prepared with AI assistance. Each source record was rechecked by PMID, with its title, journal and DOI matched against PubMed, on 2026-11-06; a record PubMed holds without a DOI was matched by title and journal. No independent professional or methodological approval is claimed. Reporting guidance supports transparent reporting and does not establish validity. The COSMIN taxonomy defines each measurement property, responsiveness among them; the fourth reference supports the missing-data plan. Source records rechecked against PubMed on 2026-11-07: record verified, confirm it supports this use. No independent professional or methodological approval is claimed.
Last source review: 2026-11-07
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.