Ask Mallard
Study design

Delphi and consensus studies

4 min read

Delphi studies use repeated anonymous ratings and controlled feedback to develop group agreement. Other consensus methods may use meetings, discussion and voting.

NHMRC EVIDENCE LEVEL · 2009Outside the NHMRC hierarchy
Why this level?not one of the five research areas this hierarchy covers. NHMRC 2009 source

Delphi and consensus studies elicit structured judgment. A treatment-effect hierarchy does not establish the quality, representativeness or correctness of that judgment.

Fit

When to use it, and when not to

Use it when

  • You need structured stakeholder judgment where evidence alone cannot settle priorities, definitions or an outcome set.
  • Patients, clinicians and other relevant groups can contribute distinct perspectives to a clearly defined question.

Not when

Before you start

What you need

Decide these first
  1. The question, stakeholders and recruitment rulesJustify expertise and lived experience. Seek balance across affected groups and report conflicts of interest.
  2. How candidate statements or outcomes are generatedCombine evidence and stakeholder input. Pilot wording and allow missing topics to be proposed.
  3. Rating scale, thresholds and stopping rulesFix consensus, exclusion and disagreement rules before ratings. State denominators and how many rounds you plan.
  4. Anonymity, feedback and any final meetingShow how feedback is summarized and which ratings people see. Explain any meeting vote separately from anonymous Delphi rounds.
It rests on these
  1. The panel covers the perspectives needed for the question.How to check: Report recruitment, expertise and stakeholder composition. Expert selection does not make a population probability sample.
  2. Panellists can express disagreement without pressure.How to check: Keep ratings anonymous when using Delphi. Balanced feedback should show disagreement as well as the group center.
  3. Dropout can create an appearance of growing agreement.How to check: Track who leaves by stakeholder group and prior ratings. Compare retained panellists with those who withdraw.
  4. Incomplete rounds require a stated missing-data rule.How to check: Missing completely at random (MCAR) means gaps are unrelated to recorded and unrecorded values; this is rarely plausible. Missing at random (MAR) means gaps depend only on recorded factors. Under MAR, consider multiple imputation; it is not automatically better than a justified complete-case analysis. Missing not at random (MNAR) is plausible if unobserved values themselves influence whether records are absent. Under MNAR, test departures with a delta or tipping-point analysis that shifts the unobserved values. Do not impute expert agreement as a substitute for a missing vote. Show sensitivity to missing dissent and the planned denominator rule.

Plain language

How it works

Define the decision and recruit the stakeholder groups whose judgment it needs. Generate and pilot candidate items. In a Delphi, participants rate items anonymously, receive controlled group feedback, and can revise their ratings in later rounds. Report modified procedures rather than assuming that every panel follows one standard recipe.

Ratings, feedback and a prespecified decision
A recruited panel rates items anonymously, receives controlled feedback and rates again. Prespecified decision rules are applied with dissent and dropout reported.Ratings, feedback and a prespecified decisionRecruit relevantstakeholder groupsAnonymous ratings andcontrolled feedbackApply fixed rules; reportdissent and dropoutA consensus result is panel judgment under a process, not proof of treatment benefit or populationagreement.
Conceptual diagram. It shows the structure of the idea, not data from any study.
Agreement needs a denominator and a context

Apply thresholds and stopping rules fixed before results. Show the number invited, responding and retained in each round, including stakeholder groups. Preserve disagreement rather than forcing a final unanimous statement. A consensus meeting can add discussion but changes the social setting of the decision. Consensus reports judgment under a process; it does not replace evidence about benefit, harm or validity.

Worked example

A worked example

Invented for illustration
Setting
A Delphi panel rates a proposed outcome for a future clinical trial.
Question
Does the item meet the threshold fixed before voting?
Approach
The rule requires at least 80 in 100 respondents to rate the item as critical. Report each stakeholder group and the round denominator.
Result
Of 50 respondents, 42 rate the item critical: 84 in 100.
What it means
The 84 in 100 figure meets the stated threshold. It describes these respondents, not universal agreement or evidence that the item improves care.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

Agreement among a selected panel is not proof that a recommendation is correct, effective or representative. Panel composition, feedback, thresholds and dropout can shape apparent consensus; report dissent and uncertainty.

  • Defining consensus after seeing which threshold retains preferred items.Instead: Register thresholds and stopping rules first. Explain deviations and show results under the original rules.
  • Calling all agreement a Delphi study.Instead: Report anonymous rounds, controlled feedback and any meeting stages. Name the actual consensus method and its modifications.
  • Reporting only a pooled percentage from an unbalanced panel.Instead: Show ratings by stakeholder group and report unresolved dissent. More clinicians should not silently outweigh patient priorities.
  • Dropping panellists with unknown ratings.Instead: This can create selection bias. Consider multiple imputation for descriptive summaries under MAR, not to manufacture consensus. Test the effect of missing dissent.

Inside Mallard

How Mallard uses this

Plans do not link to this design yet. Mallard has no dedicated sizing form for Delphi or consensus studies.

No Mallard code example points to this page.
Atlas node design.delphi, the id a plan cue records.

How to report it

Use ACCORD to report the question, protocol, stakeholder recruitment, conflicts, item generation, rating rounds, anonymity, feedback, consensus thresholds, stopping rules and deviations. State denominators, dropout, stakeholder-specific ratings and dissent. Describe any meeting separately from anonymous ratings. Find the current checklist through EQUATOR. Consensus is a result of the stated process, not proof that the agreed items are correct. A reporting guideline is not a design manual: use it to make the study transparent, not as a substitute for careful design. Check the target journal's own requirements as well, which this page cannot know.

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study (opens in a new tab) · Qual Life Res · doi 10.1007/s11136-010-9606-8
  2. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study (opens in a new tab) · Qual Life Res · doi 10.1007/s11136-018-1829-0
  3. ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi (opens in a new tab) · PLoS Med · doi 10.1371/journal.pmed.1004326
  4. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls (opens in a new tab) · BMJ · doi 10.1136/bmj.b2393
Prepared with AI assistance. Each source was verified by PMID against PubMed on 2026-11-06, matching title, authors, year and DOI. Record verification does not establish that a paper supports every use. No independent professional or methodological approval is claimed.
Last source review: 2026-11-06
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.