Reproducibility and analytic transparency
Why it matters here
The number of defensible choices in an analysis is large, and the effect of those choices on the result is often bigger than the effect being estimated. A record of what was actually run is the only way for a reader, a reviewer or the investigator six months later to know which analysis produced the figure in the abstract.
Why a finding can be false without anyone erring
The probability that a published claim is true depends on how many hypotheses the field tests, how small the studies are, how flexible the analyses are, and how strong the incentive to find something is. Under realistic values of those, a substantial fraction of published findings in some fields are expected to be false, with no fraud and no incompetence anywhere in the chain.
Practical steps, in rough order of value
- Write the analysis as a script rather than a sequence of interactive steps, so that it can be re-run from the raw data.
- Keep the data cleaning in the script too: most irreproducibility lives there, not in the model.
- Version-control the code, and record the software versions and random seeds.
- Deposit the code with the paper, and the data where consent and privacy permit.
- Report the analyses that were run and not only those that were reported, which is where prespecification and reproducibility meet.
Sharing data has limits and they are real
Individual patient data carry re-identification risk, and consent may not cover onward sharing. That is a reason for controlled access, de-identification and data-sharing agreements, not a reason to publish nothing. The code can almost always be shared even when the data cannot.
What to watch out for
Reproducibility and replication are not the same requirement. REPRODUCING means getting the same numbers from the same data and code, which is a minimum and frequently fails on ordinary papers. REPLICATING means getting the same conclusion in new data, which can legitimately fail for real reasons. Treating a failed reproduction as a scientific disagreement, or a failed replication as misconduct, confuses the two.
Clinical example
A reviewer asks how many patients were excluded for missing covariates. The answer exists only in a series of spreadsheet operations performed a year earlier and cannot now be recovered. Nothing was done wrongly, and the analysis cannot be checked.
- Why most published research findings are false (opens in a new tab) · PLoS Med · doi 10.1371/journal.pmed.0020124
- Reproducibility and transparency in biomedical sciences (opens in a new tab) · Oral Dis · doi 10.1111/odi.12588