Developing and validating a measurement instrument
Why this level?
Specify the research question: intervention, diagnostic accuracy, prognosis, aetiology, or screening intervention. Other question types are outside this hierarchy. NHMRC 2009 sourceMeasurement validation covers several properties and study designs. A coefficient or a checklist does not assign one evidence level to every measurement claim.
The question
Measurement research asks whether an instrument supports a particular interpretation and use of its scores. Define the construct, intended respondents, setting and decision before creating items or choosing psychometric analyses. A questionnaire is a delivery format; it is not evidence that the intended construct has been measured.
Develop or reuse
Search for an existing instrument with suitable content and evidence in the intended population before developing another. Check permission, burden, scoring and accessibility. Translation or adaptation can change interpretation and needs assessment rather than inheriting all evidence from the original language.
Build content validity first
Involve members of the target population and relevant experts. Explore whether the items are relevant, comprehensive and understandable, using appropriate qualitative methods and cognitive interviews. Pilot administration, response options and missing-item rules. Statistical item selection cannot recover an important part of a construct that was never represented.
Match each property to a design
- Structural validity: test whether the dimensional structure supports the intended scoring. Separate development and confirmation where feasible and justify the model.
- Internal consistency: assess it for a defensibly unidimensional scale or subscale. Alpha alone neither demonstrates unidimensionality nor covers all reliability questions.
- Reliability and measurement error: repeat measurement in participants whose underlying construct is expected to be stable; justify interval, raters and uncertainty measures.
- Construct validity: prespecify expected relationships or group differences. Criterion validity needs a defensible reference standard, not simply another convenient score.
- Responsiveness and interpretability: study meaningful change over time and its context. A statistically detectable change is not automatically clinically important.
- Measurement invariance: examine whether scores can be compared across the populations or languages for which comparisons are intended.
Example
A fatigue scale for adults receiving chemotherapy may not cover the symptoms or reading demands relevant to adolescents. The team evaluates content with adolescents before assuming the adult score and cutoffs transfer.
Record in the workspace
Record construct, target population, candidate instrument and version, content work, measurement properties, prespecified hypotheses, sampling and precision needs, scoring rules and intended comparisons. COSMIN guidance helps structure this work; no single coefficient grants validation.
What to watch out for
A high Cronbach alpha or correlation with another score does not establish validity. Evidence must support the intended construct, population, language and use.
How to report it
COSMIN reporting guideline, in its 2025 version 2.0, covers studies of the measurement properties of patient-reported outcome measures, and GRRAS covers reliability and agreement studies of any scale, instrument or classification. GRRAS was written because such reports so often left out how the sample was selected, how the study was designed and how the analysis was done, so state all three for every property you assess. A reporting guideline is not a design manual: use it to make the study transparent, not as a substitute for careful design. Check the target journal's own requirements as well, which this page cannot know.
- COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study (opens in a new tab) · Qual Life Res · doi 10.1007/s11136-018-1829-0
- The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study (opens in a new tab) · Qual Life Res · doi 10.1007/s11136-010-9606-8
- COSMIN reporting guideline for studies on measurement properties of patient-reported outcome measures: version 2.0 (opens in a new tab) · Qual Life Res · doi 10.1007/s11136-025-03950-x
- Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed (opens in a new tab) · J Clin Epidemiol · doi 10.1016/j.jclinepi.2010.03.002