Prespecification and the analysis plan
Why it changes the meaning of the result
A p-value is the probability of a result this extreme under the null, for THIS analysis, chosen in advance. If the analysis was chosen partly because it gave this result, the calculation no longer describes anything. That is true whether or not the choosing was conscious, which is why the protection has to be procedural rather than a matter of good intentions.
What the plan should fix
- The primary outcome, its exact definition and its time point.
- The population analysed, and the handling of protocol deviations.
- The primary analysis model, including covariates and how they are specified.
- The missing-data strategy and its assumption, with the sensitivity analyses.
- The subgroups, if any, and the multiplicity adjustment.
- The stopping rules and interim looks, before the first one.
Prespecification does not forbid exploration
Exploratory analysis is how findings are generated, and it is valuable. The requirement is only that it be labelled, that its results be described as hypothesis-generating, and that the confirmatory claim rest on what was planned. A paper can honestly contain both.
When the plan must change
Sometimes it must: an assumption fails, an outcome proves unmeasurable, recruitment falls short. Amend the plan, date the amendment, state whether anyone had seen outcome data, and report the change. An amendment made before unblinding and documented is a normal part of trial conduct; an undocumented one is indistinguishable from selective reporting.
What to watch out for
The problem is not deliberate manipulation; it is that a long series of small, individually reasonable choices made while looking at the data (which outcome, which time point, which covariates, which exclusions, which transformation) can produce a significant result from noise without anyone intending it. A post hoc analysis is not forbidden, and labelling it honestly costs nothing. Presenting one as though it had been planned is what invalidates the inference.
Clinical example
A trial's protocol names a six-month primary outcome. The publication reports twelve months as primary, with no amendment recorded. That single change is enough to make the reported p-value uninterpretable, and comparisons of protocols with publications find the primary outcome changed, introduced or omitted in a large minority of trials.
- SPIRIT 2013 statement: defining standard protocol items for clinical trials (opens in a new tab) · Ann Intern Med · doi 10.7326/0003-4819-158-3-201302050-00583
- Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles (opens in a new tab) · JAMA · doi 10.1001/jama.291.20.2457