Meta-analysis models and heterogeneity
What question does this help answer?
What is the average effect across studies, how much do the true effects vary between them, and what does that variation imply for a new setting?
When to use it
Use it within a systematic review when studies are similar enough in population, intervention, comparator, outcome and timing that combining them answers a meaningful question, and when each study's estimate and its uncertainty are available on a common scale.
When not to use it
Do not pool studies that measure different things, and do not pool results at high risk of bias with others without examining what that does to the summary. A structured narrative synthesis is the honest result when studies cannot be combined, and pooling a handful of very different studies can produce a precise-looking number that describes none of them.
Two models, two questions
- A common-effect (fixed-effect) model assumes every study estimates the same true effect, and its summary is that effect. It is defensible when studies are close replicates.
- A random-effects model assumes true effects vary across studies around an average, and its summary is that average. It is the usual choice when clinical or methodological diversity is expected.
- Choose the model in the protocol from how similar the studies are expected to be, not afterwards from which model gives a narrower interval or a significant result.
- Random-effects weights are more equal across studies than common-effect weights, so small studies have relatively more influence on the summary.
One true effect, or a distribution of them
What a forest plot of studies and their summary looks like
Reading heterogeneity
- The between-study variance (tau-squared) is the amount of variation in true effects, on the scale of the effect measure.
- I-squared is the percentage of observed variation not attributable to chance. It depends on the size of the studies and does not say how different the effects are.
- A non-significant test for heterogeneity does not show consistency, because the test has little power when studies are few.
- A prediction interval shows where the true effect in a new, similar setting is likely to lie, and it is often much wider than the confidence interval for the average.
Uncertainty when studies are few
With few studies, the between-study variance is estimated imprecisely. The traditional DerSimonian-Laird approach can then give confidence intervals that are too narrow; the Hartung-Knapp-Sidik-Jonkman adjustment usually gives more appropriate coverage, although no method performs well with very few studies. Name the variance estimator and the interval method so readers can judge this.
Explaining differences between studies
Prespecify a small number of subgroup analyses or meta-regressions based on plausible reasons for different effects. These compare studies rather than patients, so they are observational, open to confounding by other study features and to ecological bias, and they need many studies to be informative.
Clinical example
Eight trials compare a physiotherapy programme with usual care for length of stay. The protocol anticipated differences in programme intensity, so it plans a random-effects model, names the variance estimator and interval method, commits to reporting a prediction interval, and prespecifies one subgroup by intensity. No pooled result is shown; the example illustrates planning choices only.
What to watch out for
A random-effects summary is the average of effects that differ, not the effect in any particular setting, and its confidence interval does not say where the effect in a new study is likely to fall. I-squared is a proportion rather than an amount of heterogeneity, and with few studies the between-study variance is estimated very imprecisely, so choosing a model because it gives the preferred answer is not a model choice.
Before using this in your workspace
Record the effect measure, the model and the reason for it, the between-study variance estimator and confidence interval method, whether a prediction interval will be reported, the prespecified heterogeneity analyses, and how studies at high risk of bias will be handled.
- Interpretation of random effects meta-analyses (opens in a new tab) · BMJ · doi 10.1136/bmj.d549
- A re-evaluation of random-effects meta-analysis (opens in a new tab) · J R Stat Soc Ser A Stat Soc · doi 10.1111/j.1467-985X.2008.00552.x
- The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method (opens in a new tab) · BMC Med Res Methodol · doi 10.1186/1471-2288-14-25
- Plea for routinely presenting prediction intervals in meta-analysis (opens in a new tab) · BMJ Open · doi 10.1136/bmjopen-2015-010247