Confidence intervals and p-values
What question does this help answer?
“How large might the effect reasonably be, how precise is the estimate, and how strongly do the data conflict with a null hypothesis?”
Why it matters
Research decisions should not collapse to “p < 0.05” versus “not significant.” Magnitude, direction, uncertainty, clinical importance, design validity, and prior evidence all matter.
How Mallard may use this concept
Mallard may remind users to report effect estimates with confidence intervals and avoid interpreting nonsignificance as proof of no effect.
How to think about it
A 95% confidence interval is not literally a 95% probability statement about a fixed parameter under standard frequentist interpretation. It reflects a procedure that would capture the true parameter in 95% of repeated samples under the model assumptions. A p-value does not give the probability that the null hypothesis is true.
An interval against what matters
What to watch out for
- Statistical significance is not the same as clinical importance.
- A wide confidence interval may include both clinically important benefit and harm.
- A small p-value does not repair bias, confounding, poor measurement, or data dredging.
- A p-value above 0.05 does not demonstrate equivalence or no effect.
- Precision typically improves with more information, not with more significance testing.
Clinical example
An adjusted risk ratio of 0.82 with a 95% CI of 0.60 to 1.12 is compatible with a potentially meaningful reduction and with little or no reduction; calling the result simply “negative” loses important information.
- How to obtain the confidence interval from a P value (opens in a new tab) · BMJ · doi 10.1136/bmj.d2090
- Absence of evidence is not evidence of absence (opens in a new tab) · BMJ · doi 10.1136/bmj.311.7003.485