Ask Mallard
Core concept

Choosing a test for means

5 min read

Start with pairing and the number of groups, then check spread and whether the question needs an adjusted mean difference.

Fit

When it matters, and when it does not

It matters when

  • You want to compare the average of a continuous outcome.
  • You must choose a paired test, an independent-groups test or a comparison across several groups.

It matters less when

Before you start

What to check in your study

Decide these first
  1. Who contributes each valuerepeatedMeasuresRepeated values from the same person form a pair, not independent groups.
  2. How many groupsSpecify the planned contrast before looking at results.
  3. The mean difference of interestsummaryMeasureDecide whether baseline values or other factors need adjustment.
  4. With three or more groups, the pairs you will compareName them in advance, with a method that adjusts for making several comparisons. Tukey's suits groups with a similar spread. When the spreads differ, use Games-Howell.
It rests on these
  1. People or other independent units are sampled separately.How to check: Paired tests allow dependence within pairs. Clinic-level clustering needs a separate analysis plan.
  2. The mean and its uncertainty are meaningful.How to check: Inspect outliers and skew. Small samples with heavy tails may need a robust mean method or a justified transformation.
  3. Student's t-test and classic ANOVA assume the groups have a similar spread.How to check: Use Welch's t-test by default. It drops this assumption and costs almost nothing when spreads are equal. Welch's ANOVA does the same for three or more groups, and Games-Howell then compares the pairs.

Plain language

How it works

Why multiple comparisons matter
The chance of at least one false positive rises with the number of independent tests each run at 0.05: 5 percent for one test, 23 percent for five, 40 percent for ten, and 64 percent for twenty.Chance of at least one false positive, with every null hypothesis true151015200%25%50%75%100%Number of independent tests, each at 0.05Chance of at least one false positive5%23%40%64%Calculated as one minus 0.95 raised to the number of tests, which assumes the tests are independent.
Worked arithmetic. The inputs are round illustrative numbers, not results from any study; the outputs are calculated from them.
How it works

For independent groups, Welch compares two means without assuming equal variances. A paired t-test compares the mean of differences within matched pairs. Its normality assumption concerns those differences.

The choice at a glance
Independent groups: Two unrelated groups leads to Welch t-test; Paired values: One difference within each pair leads to Paired t-test; Several groups: Independent groups and mean contrasts leads to ANOVA or Welch ANOVA.Independent groupsTwo unrelated groupsWelch t-testUse the study question and assumptions together.Paired valuesOne difference within each pairPaired t-testUse the study question and assumptions together.Several groupsIndependent groups and mean contrastsANOVA or Welch ANOVAUse the study question and assumptions together.
Conceptual diagram. It shows the structure of the idea, not data from any study.
Reading the choice

One-way analysis of variance, or ANOVA, tests equality of means across independent groups. Ordinary ANOVA assumes equal variances; Welch ANOVA relaxes that assumption. Neither accounts for repeated observations or clustering by itself.

What decides it

For a trial with baseline and follow-up values, consider baseline-adjusted analysis of covariance, called ANCOVA. Rank tests can be useful for other targets, but do not automatically test a difference in means.

Worked example

A worked example

Invented for illustration
Setting
Four patients have pain scores before and after care.
Question
Should the team treat the two sets of scores as independent?
Approach
Before scores are 8, 7, 6 and 9. After scores for those same patients are 5, 5, 4 and 8.
Result
The mean falls from 7.5 to 5.5 points, a change of -2 points.
What it means
Use a paired comparison of within-person differences. The paired t-test uses the spread of those differences, not the two groups as unrelated samples.

In practice

Recent studies PubMed search · not vetted

Searching PubMed…

Watch out

Common mistakes

The one to remember

Choose the test for the design and the mean difference you want. A normality test alone cannot make that choice.

  • Using a pooled t-test because a variance test was not significant.Instead: Welch handles unequal group variances. Do not let a preliminary variance test choose the main analysis.
  • Treating an overall analysis of variance result as every pairwise difference.Instead: ANOVA tests whether all group means are equal. State planned contrasts and handle multiple comparisons.

Inside Mallard

How Mallard uses this

Mallard lists Choosing a test for means under Core research concepts.

No Mallard code example points to this page.
Calculators: Independent t-test (Welch) (§2.3), Mann–Whitney U (§2.4), Paired t-test (§2.5), Wilcoxon signed-rank (§2.6), One-way ANOVA (§3.1), Kruskal–Wallis (§3.2).
Atlas node concept.choosing_means_test, the id a plan cue records.

Work it out

Related Learn pages

Related designs and methods

Evidence

Sources & further reading

  1. Consequences of choosing samples in hypothesis testing to ensure homogeneity of variance (opens in a new tab) · Br J Math Stat Psychol · doi 10.1111/bmsp.12001
  2. Optimal sample size allocation for Welch's test in one-way heteroscedastic ANOVA (opens in a new tab) · Behav Res Methods · doi 10.3758/s13428-014-0477-8
  3. Best practice in statistics: Use the Welch t-test when testing the difference between two groups (opens in a new tab) · Ann Clin Biochem · doi 10.1177/0004563221992088
Prepared with AI assistance. Source titles, authors, publication years and available DOIs were verified against PubMed on 2026-11-06. No independent professional approval is claimed. Zimmerman studies the harm of selecting samples to achieve equal variances. Shieh and Jan develop allocation methods for Welch ANOVA. West explains the two-group Welch choice.
Last source review: 2026-11-06
AI-generated educational content
Mallard uses reputable methodological and reporting sources, but this material may contain errors or omit important context. Verify important methodological decisions with the cited sources and, when appropriate, a qualified research methodologist or statistician.