Choosing a test for means
Start with pairing and the number of groups, then check spread and whether the question needs an adjusted mean difference.
Fit
When it matters, and when it does not
It matters when
- You want to compare the average of a continuous outcome.
- You must choose a paired test, an independent-groups test or a comparison across several groups.
It matters less when
- You want to compare ranks rather than means. Consider a rank-based test. Read: Mann-Whitney U test
- Patients have several follow-up measurements. Consider a longitudinal model. Read: Choosing an approach for repeated measures
Before you start
What to check in your study
- Who contributes each valuerepeatedMeasuresRepeated values from the same person form a pair, not independent groups.
- How many groupsSpecify the planned contrast before looking at results.
- The mean difference of interestsummaryMeasureDecide whether baseline values or other factors need adjustment.
- With three or more groups, the pairs you will compareName them in advance, with a method that adjusts for making several comparisons. Tukey's suits groups with a similar spread. When the spreads differ, use Games-Howell.
- People or other independent units are sampled separately.How to check: Paired tests allow dependence within pairs. Clinic-level clustering needs a separate analysis plan.
- The mean and its uncertainty are meaningful.How to check: Inspect outliers and skew. Small samples with heavy tails may need a robust mean method or a justified transformation.
- Student's t-test and classic ANOVA assume the groups have a similar spread.How to check: Use Welch's t-test by default. It drops this assumption and costs almost nothing when spreads are equal. Welch's ANOVA does the same for three or more groups, and Games-Howell then compares the pairs.
Plain language
How it works
For independent groups, Welch compares two means without assuming equal variances. A paired t-test compares the mean of differences within matched pairs. Its normality assumption concerns those differences.
One-way analysis of variance, or ANOVA, tests equality of means across independent groups. Ordinary ANOVA assumes equal variances; Welch ANOVA relaxes that assumption. Neither accounts for repeated observations or clustering by itself.
For a trial with baseline and follow-up values, consider baseline-adjusted analysis of covariance, called ANCOVA. Rank tests can be useful for other targets, but do not automatically test a difference in means.
Worked example
A worked example
- Setting
- Four patients have pain scores before and after care.
- Question
- Should the team treat the two sets of scores as independent?
- Approach
- Before scores are 8, 7, 6 and 9. After scores for those same patients are 5, 5, 4 and 8.
- Result
- The mean falls from 7.5 to 5.5 points, a change of -2 points.
- What it means
- Use a paired comparison of within-person differences. The paired t-test uses the spread of those differences, not the two groups as unrelated samples.
In practice
Recent studies PubMed search · not vetted
Watch out
Common mistakes
The one to remember
Choose the test for the design and the mean difference you want. A normality test alone cannot make that choice.
- Using a pooled t-test because a variance test was not significant.Instead: Welch handles unequal group variances. Do not let a preliminary variance test choose the main analysis.
- Treating an overall analysis of variance result as every pairwise difference.Instead: ANOVA tests whether all group means are equal. State planned contrasts and handle multiple comparisons.
Inside Mallard
How Mallard uses this
Mallard lists Choosing a test for means under Core research concepts.
concept.choosing_means_test, the id a plan cue records.Work it out
Related Learn pages
Related designs and methods
Evidence
Sources & further reading
- Consequences of choosing samples in hypothesis testing to ensure homogeneity of variance (opens in a new tab) · Br J Math Stat Psychol · doi 10.1111/bmsp.12001
- Optimal sample size allocation for Welch's test in one-way heteroscedastic ANOVA (opens in a new tab) · Behav Res Methods · doi 10.3758/s13428-014-0477-8
- Best practice in statistics: Use the Welch t-test when testing the difference between two groups (opens in a new tab) · Ann Clin Biochem · doi 10.1177/0004563221992088