Delphi and consensus studies
Delphi studies use repeated anonymous ratings and controlled feedback to develop group agreement. Other consensus methods may use meetings, discussion and voting.
Why this level?
not one of the five research areas this hierarchy covers. NHMRC 2009 sourceDelphi and consensus studies elicit structured judgment. A treatment-effect hierarchy does not establish the quality, representativeness or correctness of that judgment.
Fit
When to use it, and when not to
Use it when
- You need structured stakeholder judgment where evidence alone cannot settle priorities, definitions or an outcome set.
- Patients, clinicians and other relevant groups can contribute distinct perspectives to a clearly defined question.
Not when
- You need to estimate what the population believes. Use a survey with a sampling design suited to that population. Read: Survey and questionnaire research
- You need to show that a treatment works. Use a comparative clinical study, such as a randomized trial. Read: Randomized controlled trial
Before you start
What you need
- The question, stakeholders and recruitment rulesJustify expertise and lived experience. Seek balance across affected groups and report conflicts of interest.
- How candidate statements or outcomes are generatedCombine evidence and stakeholder input. Pilot wording and allow missing topics to be proposed.
- Rating scale, thresholds and stopping rulesFix consensus, exclusion and disagreement rules before ratings. State denominators and how many rounds you plan.
- Anonymity, feedback and any final meetingShow how feedback is summarized and which ratings people see. Explain any meeting vote separately from anonymous Delphi rounds.
- The panel covers the perspectives needed for the question.How to check: Report recruitment, expertise and stakeholder composition. Expert selection does not make a population probability sample.
- Panellists can express disagreement without pressure.How to check: Keep ratings anonymous when using Delphi. Balanced feedback should show disagreement as well as the group center.
- Dropout can create an appearance of growing agreement.How to check: Track who leaves by stakeholder group and prior ratings. Compare retained panellists with those who withdraw.
- Incomplete rounds require a stated missing-data rule.How to check: Missing completely at random (MCAR) means gaps are unrelated to recorded and unrecorded values; this is rarely plausible. Missing at random (MAR) means gaps depend only on recorded factors. Under MAR, consider multiple imputation; it is not automatically better than a justified complete-case analysis. Missing not at random (MNAR) is plausible if unobserved values themselves influence whether records are absent. Under MNAR, test departures with a delta or tipping-point analysis that shifts the unobserved values. Do not impute expert agreement as a substitute for a missing vote. Show sensitivity to missing dissent and the planned denominator rule.
Plain language
How it works
Define the decision and recruit the stakeholder groups whose judgment it needs. Generate and pilot candidate items. In a Delphi, participants rate items anonymously, receive controlled group feedback, and can revise their ratings in later rounds. Report modified procedures rather than assuming that every panel follows one standard recipe.
Apply thresholds and stopping rules fixed before results. Show the number invited, responding and retained in each round, including stakeholder groups. Preserve disagreement rather than forcing a final unanimous statement. A consensus meeting can add discussion but changes the social setting of the decision. Consensus reports judgment under a process; it does not replace evidence about benefit, harm or validity.
Worked example
A worked example
- Setting
- A Delphi panel rates a proposed outcome for a future clinical trial.
- Question
- Does the item meet the threshold fixed before voting?
- Approach
- The rule requires at least 80 in 100 respondents to rate the item as critical. Report each stakeholder group and the round denominator.
- Result
- Of 50 respondents, 42 rate the item critical: 84 in 100.
- What it means
- The 84 in 100 figure meets the stated threshold. It describes these respondents, not universal agreement or evidence that the item improves care.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
Agreement among a selected panel is not proof that a recommendation is correct, effective or representative. Panel composition, feedback, thresholds and dropout can shape apparent consensus; report dissent and uncertainty.
- Defining consensus after seeing which threshold retains preferred items.Instead: Register thresholds and stopping rules first. Explain deviations and show results under the original rules.
- Calling all agreement a Delphi study.Instead: Report anonymous rounds, controlled feedback and any meeting stages. Name the actual consensus method and its modifications.
- Reporting only a pooled percentage from an unbalanced panel.Instead: Show ratings by stakeholder group and report unresolved dissent. More clinicians should not silently outweigh patient priorities.
- Dropping panellists with unknown ratings.Instead: This can create selection bias. Consider multiple imputation for descriptive summaries under MAR, not to manufacture consensus. Test the effect of missing dissent.
Inside Mallard
How Mallard uses this
Plans do not link to this design yet. Mallard has no dedicated sizing form for Delphi or consensus studies.
design.delphi, the id a plan cue records.How to report it
Use ACCORD to report the question, protocol, stakeholder recruitment, conflicts, item generation, rating rounds, anonymity, feedback, consensus thresholds, stopping rules and deviations. State denominators, dropout, stakeholder-specific ratings and dissent. Describe any meeting separately from anonymous ratings. Find the current checklist through EQUATOR. Consensus is a result of the stated process, not proof that the agreed items are correct. A reporting guideline is not a design manual: use it to make the study transparent, not as a substitute for careful design. Check the target journal's own requirements as well, which this page cannot know.
Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study (opens in a new tab) · Qual Life Res · doi 10.1007/s11136-010-9606-8
- COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study (opens in a new tab) · Qual Life Res · doi 10.1007/s11136-018-1829-0
- ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi (opens in a new tab) · PLoS Med · doi 10.1371/journal.pmed.1004326
- Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls (opens in a new tab) · BMJ · doi 10.1136/bmj.b2393