Choosing Bayesian or frequentist inference
Bayesian inference gives probabilities conditional on a prior and model. Frequentist inference describes procedures through their behavior across repeated samples.
Fit
When it matters, and when it does not
It matters when
- You must decide whether a posterior probability or a frequentist interval and test will answer the research question.
- Earlier evidence may inform a prior, and its relevance and influence need explicit scrutiny.
It matters less when
- The model ignores clustering, missing data or confounding. Address those design problems first; either inference framework can give misleading answers. Read: The estimand
- The aim is to obtain a favorable conclusion after seeing results. Follow the prespecified analysis and label any reanalysis. Read: Prespecification and the analysis plan
Before you start
What to check in your study
- The outcome and its summary measureoutcomeTypesummaryMeasureSuch as an odds ratio for a yes-or-no outcome. The prior is set on the same scale.
- The main prior, and where it comes fromEarlier trials, expert opinion or a deliberately skeptical choice. Write down its center and spread before seeing the data.
- Other priors for a sensitivity analysisAt least a skeptical prior centered on no effect and a vague one. The range of answers across them is the honest result.
- The effect that matters, and how sure you must beSuch as a 90 in 100 chance that the risk falls by at least 2 in 100. Fix both in advance.
- For a trial, how its decision rule behavesSimulate how often it would declare benefit when there is none, and how often it would miss a real effect.
- The model for the data is right, just as in any other analysis.How to check: A prior cannot fix a wrong outcome model, missing confounders or ignored clustering.
- Evidence borrowed into the prior applies to these patients.How to check: Explain why earlier trials match this population, treatment and outcome. Give them less weight if they may not.
- The computer's sampling has settled on the posterior.How to check: Most models are fitted by simulation, called MCMC. Check that several chains agree before reading any result.
Plain language
How it works
Bayesian inference combines a stated prior with the data through a model. Its posterior describes uncertainty conditional on those choices. Frequentist inference evaluates a procedure across repeated samples under stated assumptions. It can provide confidence intervals and tests without assigning a probability to a fixed effect.
A 95% credible interval holds 95% posterior probability given the prior, model and data. A 95% confidence procedure covers the fixed true effect in 95% of repeated samples under its assumptions. A P value is not the probability of benefit. Both approaches require a suitable model and honest sensitivity checks. Prespecify priors, decision thresholds and trial operating characteristics rather than choosing a favorable framework afterwards.
Worked example
A worked example
- Setting
- A small randomized trial in 200 adults with severe pneumonia, 100 given a new treatment and 100 usual care.
- Question
- How likely is it that the new treatment lowers death within 30 days?
- Approach
- A Bayesian analysis of the risk ratio, on the log scale with a normal approximation. Two priors were fixed in advance: a vague one, and a skeptical one centered on no effect and carrying as much weight as the trial.
- Result
- 24 of 100 died with the new treatment and 30 of 100 with usual care: a risk difference of -6 per 100 and a risk ratio of 0.8.
- What it means
- Under the vague prior, the probability that the treatment lowers deaths is about 83 in 100. Under the skeptical prior, it is about 75 in 100. Neither is a P value, and neither settles the question.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
The prior is a modeling choice with real consequences, and a Bayesian analysis is only as credible as its justification and the sensitivity analysis around it: reporting one posterior from one prior chosen without discussion is not more honest than a p-value, it is less. A credible interval is also not a confidence interval renamed. And Bayesian methods are not a way to rescue a null trial: computing a high probability of benefit from an enthusiastic prior after the fact is advocacy, not analysis.
- Reading a P value as the probability that the treatment works.Instead: Only a posterior probability answers that question, and it depends on the prior.
- Calling a flat prior non-informative.Instead: A prior flat on one scale is not flat on another, and it can favor implausibly large effects. Use a weakly informative prior and say why.
- Assuming a Bayesian trial needs no control of false positives.Instead: Regulators still ask how often the decision rule would declare benefit wrongly. Estimate it by simulation before the trial starts.
- Reporting only the probability of any benefit.Instead: Also give the probability of a benefit large enough to matter, and of harm. A tiny effect can be very probable and still unimportant.
Inside Mallard
How Mallard uses this
For a causal question with a binary outcome and an odds ratio, Mallard may offer Bayesian logistic regression. Its method link opens the inference concept; the concept does not fit a model.
bayesian-logistic-regression).concept.choosing_bayesian_frequentist, the id a plan cue records.Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Bayesian methodology for the design and interpretation of clinical trials in critical care medicine: a primer for clinicians (opens in a new tab) · Crit Care Med · doi 10.1097/CCM.0000000000000576
- A Gentle Introduction to the Comparison Between Null Hypothesis Testing and Bayesian Analysis: Reanalysis of Two Randomized Controlled Trials (opens in a new tab) · J Med Internet Res · doi 10.2196/10873