Risk of bias and certainty of evidence
What question does this help answer?
For each result that matters, could the way the study was designed, conducted or analysed have pushed it away from the truth? And across the studies, how sure can anyone be about the effect?
When to use it
Assess risk of bias for each included study in a systematic review, and when appraising a key study that will shape a protocol, guideline or clinical decision. Rate certainty when summarizing evidence for an outcome to inform a decision.
What it is not
Risk of bias is not reporting quality: a well-reported study can be at high risk of bias, and a poorly reported one may simply leave bias unclear. It is also not imprecision, which concerns random error and the amount of data.
Use a tool that matches the design
- Randomized trials: RoB 2 assesses bias from the randomization process, deviations from intended interventions, missing outcome data, outcome measurement and selection of the reported result, for a specific result.
- Non-randomized studies of interventions: ROBINS-I judges bias against a hypothetical target trial, including confounding and selection of participants into the study.
- Diagnostic accuracy studies: QUADAS-2 covers patient selection, the index test, the reference standard, and flow and timing.
- Each tool reaches a judgement for each domain and an overall judgement through its own guidance. Adding items into a numerical score is not how these tools work.
From studies to certainty
GRADE rates the certainty of evidence for each outcome as high, moderate, low or very low. Evidence from randomized trials usually starts higher than evidence from non-randomized studies. Certainty can be rated down for risk of bias, inconsistency, indirectness, imprecision and publication bias, and in specific circumstances rated up. The rating is a transparent judgement, recorded with its reasons, rather than a calculation.
Clinical example
A review of a fall-prevention programme includes six trials. Most are at low risk of bias from randomization but at high risk for outcome measurement, because falls were self-reported by participants who knew their allocation. The review records that judgement per trial, carries it into the certainty rating for the falls outcome, and states why certainty was rated down. This example is illustrative.
What to watch out for
Risk of bias is judged for a specific result, domain by domain, not for a journal, a design label or a study as a whole, and it is not the same as reporting quality. Adding checklist items into a score hides which bias matters, and a certainty rating is a structured judgement about a body of evidence for one outcome, not a property of a single study.
Before using this in your workspace
Name the tool for each included design in the protocol, who will assess each study and how disagreements will be resolved, which results are assessed, and how the judgements will feed the synthesis and the certainty ratings.
- RoB 2: a revised tool for assessing risk of bias in randomised trials (opens in a new tab) · BMJ · doi 10.1136/bmj.l4898
- ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions (opens in a new tab) · BMJ · doi 10.1136/bmj.i4919
- QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies (opens in a new tab) · Ann Intern Med · doi 10.7326/0003-4819-155-8-201110180-00009
- GRADE guidelines: 3. Rating the quality of evidence (opens in a new tab) · J Clin Epidemiol · doi 10.1016/j.jclinepi.2010.07.015