Sensitivity analyses
A sensitivity analysis repeats the main analysis under other plausible assumptions or choices, to show whether the study's conclusion depends on them.
Fit
When it matters, and when it does not
It matters when
- The conclusion rests on an assumption the data cannot check, such as no unmeasured confounding or values missing at random.
- A reasonable analyst could have chosen differently, such as the exposure window, the outcome definition or the confounders.
- Outcomes are missing, and the patients without them may differ in ways the data do not show.
It matters less when
- You are searching for the analysis that gives the hoped-for result. Prespecify a few analyses aimed at the study's real weak points. Read: Prespecification and the analysis plan
- A large bias is foreseeable, such as sicker patients getting the treatment. Reduce it by design, such as an active comparator, not only afterwards. Read: Target trial emulation
Before you start
What to check in your study
- The study's main threats to validitySuch as unmeasured confounding, missing outcomes or a debatable exposure window. Each sensitivity analysis targets one.
- The alternatives, and their range, set in advanceName each alternative assumption or choice, and the values it will take, in the protocol or analysis plan.
- Why outcomes may be missingIf missing not at random is plausible, plan a delta-adjusted or tipping-point analysis around the missing-at-random analysis.
- Confounders that were not measuredassignmentName them and what is known of their strength, so an E-value or a bias analysis can be judged against something real.
- What would count as the conclusion changingSuch as the interval crossing no effect, or the estimate becoming smaller than an important size. Decide before seeing the results.
- The alternatives tried cover the plausible departures.How to check: A sensitivity analysis is only as wide as its scenarios. Justify the range from clinical knowledge or earlier studies.
- The main analysis is credible under its own assumptions.How to check: A tipping-point analysis starts from a credible missing-at-random analysis. Sensitivity analyses cannot rescue a flawed main one.
- Each sensitivity analysis answers the same question as the main one.How to check: If it changes the question, such as dropping patients who stopped treatment, a different answer is not a lack of robustness.
Plain language
How it works
List the assumptions most able to change the answer. Choose plausible alternatives before seeing results, then compare estimates and intervals. Keep the scientific question fixed. A change of target population or treatment strategy answers a different question, which is a supplementary analysis.
An E-value benchmarks unmeasured confounding on a risk-ratio scale. A delta-adjusted tipping-point analysis explores departures from assumptions about missing outcomes. Their single-method articles explain the calculations. Neither addresses every source of bias. Other threats may call for different exposure definitions, lag windows or model forms.
Worked example
A worked example
- Setting
- Health records of 600 older adults: 300 who started a sedating sleep drug and 300 who did not.
- Question
- How much does the starting association depend on assumptions about confounding?
- Approach
- Compare the starting risk estimate with prespecified, plausible departures from its assumptions. Keep the target question fixed and compare estimates and intervals.
- Result
- 54 of 300 drug users had such a fall, against 30 of 300 non-users. That is a risk difference of 8 per 100 and a risk ratio of 1.8, with a 95% confidence interval of 1.19 to 2.73.
- What it means
- The starting comparison alone cannot establish robustness. Unmeasured confounding and missing outcomes call for different sensitivity methods, with clinically justified ranges of assumptions.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
“Results were robust” should mean the important scientific conclusion remained reasonably stable across prespecified plausible alternatives, not merely that every p-value stayed below 0.05.
- Reading an E-value as the chance that confounding explains the result.Instead: It gives the strength a confounder would need. Whether one that strong exists is a judgment about the subject.
- Offering a complete-case analysis as the sensitivity analysis for missing data.Instead: It makes its own assumption rather than exploring departures from missing at random. Use a delta-adjusted or tipping-point analysis.
- Presenting a per-protocol analysis as a sensitivity analysis beside intention to treat.Instead: It estimates a different effect. Label it a supplementary analysis, not a test of robustness.
Inside Mallard
How Mallard uses this
When a plan names sensitivity analysis, an E-value, a negative control or a tipping point, its concept cue can link to sensitivity analyses. Specific methods have separate Learn articles.
concept.sensitivity, the id a plan cue records.Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Principles of confounder selection (opens in a new tab) · Eur J Epidemiol · doi 10.1007/s10654-019-00494-6
- Methodological Tutorial Series for Epidemiological Studies: Confounder Selection and Sensitivity Analyses to Unmeasured Confounding From Epidemiological and Statistical Perspectives (opens in a new tab) · J Epidemiol · doi 10.2188/jea.JE20240082
- Sensitivity Analysis in Observational Research: Introducing the E-Value (opens in a new tab) · Ann Intern Med · doi 10.7326/M16-2607
- Sensitivity analysis for clinical trials with missing continuous outcome data using controlled multiple imputation: A practical guide (opens in a new tab) · Stat Med · doi 10.1002/sim.8569
- Negative controls: a tool for detecting confounding and bias in observational studies (opens in a new tab) · Epidemiology · doi 10.1097/EDE.0b013e3181d61eeb