Choosing a design to evaluate a QI change
A quality improvement study tests a change in how care is delivered, and reports what changed, in which setting, and how strongly the evidence links the change to the results.
Fit
When it matters, and when it does not
It matters when
- A service has a measured gap in care, and a change to close it is ready to test.
- You can measure the outcome, the process and any harm repeatedly, before and after the change.
- You want to learn whether the change worked here, and why, while adapting it as you go.
It matters less when
- You want to show that the change works beyond your service. Plan a comparison able to show it, such as an interrupted time series. Read: Interrupted time series
- The change can be randomized between wards or sites. Use a cluster randomized trial. Read: Cluster randomized trial
Before you start
What to check in your study
- The aim, with a number and a dateSuch as cutting discharge medicine errors from 30 to 15 in 100 by June. A clear aim makes progress measurable.
- Outcome, process and balancing measuresoutcomeTypeThe outcome that matters, whether the change was delivered, and any harm it caused elsewhere. Define each one's numerator and denominator.
- The evaluation designassignmentrepeatedMeasuresA single before and after describes a change. Many measurements over time, or a comparison ward, can show more.
- What else is changingStaffing, case mix, coding or other projects. Record them with dates, because each can explain a change.
- A missing-data plan that starts from an assumptionDecide why audits or records may be missing in each period before choosing a method. The assumption picks the method.
- The expected timing and shape of the change are stated in advance.How to check: Date the change and each adaptation. A design chosen after seeing the run chart can find a pattern in noise.
- Nothing else changed at the same time.How to check: Look for other projects, new staff, a change in patients and regression to the mean after a bad month.
- The analysis allows for repeated measurements and for wards.How to check: Points close in time are alike, and patients on one ward are alike. Few wards give little precision, however many patients.
- Missing audits and records follow the assumption you named.How to check: Audits are skipped more often on busy days, so missing completely at random (MCAR) is rarely plausible. Missing at random (MAR) means the gaps depend only on recorded factors, such as ward and day. Under MAR, multiple imputation can help, but it is not automatically better than analyzing the audits done. If audits are skipped when care went worst, missing not at random (MNAR) is plausible. Then add a tipping-point analysis, which shifts the missing results until the conclusion changes.
Plain language
How it works
Name the care gap, the change and the outcome before choosing a design. Include process measures, such as checklist use, and balancing measures, such as discharge delays. Record other changes that could explain the result.
One measurement before and after describes change but leaves its cause uncertain. Repeated points with a clear intervention date support an interrupted time series. A concurrent comparison can help separate shared changes. Random allocation strengthens the comparison when feasible. Use the linked pages for each design and chart.
Worked example
A worked example
- Setting
- A medical ward introduces a pharmacist discharge checklist, with weekly audits of discharge medicine errors.
- Question
- Did errors fall after the checklist started?
- Approach
- Twenty weeks of audits before the change and twenty after are plotted, with checklist use and discharge delays measured too. A pharmacy staffing change in week 25 is recorded.
- Result
- Errors fell from 30 in 100 discharges before the checklist to 18 in 100 after, a fall of 12 per 100.
- What it means
- Errors fell, but the staffing change could explain part of the fall. A comparison ward, or a longer series, would show more about the cause.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
An improvement after implementation does not by itself show that the intervention caused it. Secular trends, regression to the mean, changing denominators and concurrent changes need an explicit evaluation strategy.
- Calling a fall after the change proof that the change worked.Instead: Secular trends, regression to the mean and other changes can all produce a fall. Choose an evaluation that can rule them out.
- Reporting only the process measure.Instead: A checklist can be completed every time while errors stay the same. Report outcome, process and balancing measures together.
- Using the label quality improvement to skip an ethics or oversight decision.Instead: Ask your institution which review applies, especially if you will publish or compare groups.
Inside Mallard
How Mallard uses this
Plans do not link to this concept yet. If your plan types its data as clustered, such as by ward, and is sized as if patients were independent, Mallard withholds the headline sample size.
concept.choosing_qi_design, the id a plan cue records.Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process (opens in a new tab) · BMJ Qual Saf · doi 10.1136/bmjqs-2015-004411
- Interrupted time series regression for the evaluation of public health interventions: a tutorial (opens in a new tab) · Int J Epidemiol · doi 10.1093/ije/dyw098
- How to study improvement interventions: a brief overview of possible study types (opens in a new tab) · BMJ Qual Saf · doi 10.1136/bmjqs-2014-003620
- Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls (opens in a new tab) · BMJ · doi 10.1136/bmj.b2393