How to choose a statistical test for two groups

Two questions decide almost everything: are the groups independent or paired, and what kind of variable is the outcome? Settle both before looking at any p values. Choosing the test after seeing which one gives a significant result inflates the false-positive rate.

Groups are independent when each observation comes from a different, unrelated unit: students in two different classes, patients randomised to two treatments, plants grown under two conditions.

Data are paired when each value in one set is naturally linked to one value in the other: the same participants before and after a course, left and right eyes of the same people, or participants deliberately matched on age and sex. A paired analysis works on the differences within each pair, which removes the variation between individuals and usually gives a more precise comparison.

Getting this wrong matters. Analysing paired data as independent ignores the pairing and usually throws away power. Analysing independent data as paired is simply invalid, because there is no meaningful pairing to take differences of.

Student's t-test#

The classic two-sample t-test compares the means of two groups. It assumes the observations are independent, the outcome is roughly normally distributed within each group (or the samples are large enough for the sampling distribution of the mean difference to be close to normal), and the two groups have equal population variances. It pools the two variances into one estimate.

Welch's t-test#

Welch's t-test also compares means and makes the same assumptions except one: it does not assume equal variances. It estimates the standard error from each group's variance separately:

The degrees of freedom come from the Welch–Satterthwaite approximation and are usually not a whole number. That is expected; report them to one or two decimal places.

Mann–Whitney U test#

The Mann–Whitney U test (also called the Wilcoxon rank-sum test) works on ranks instead of raw values. It is the natural choice for ordinal outcomes, such as Likert-type ratings, and is robust to outliers. It does not need normality.

Be careful how you describe what it tests. In general it tests whether values in one group tend to be larger than values in the other (whether the probability that a random value from group A exceeds a random value from group B differs from one half). It is a test of a difference in medians only if you also assume both distributions have the same shape and spread. It is not a drop-in replacement for the t-test when variances differ: unequal spreads affect it too.

When the two groups really do have equal variances, Welch's and Student's tests give almost identical results. When they do not, Student's test can be badly wrong, especially when the group sizes also differ:

• If the larger group has the larger variance, Student's test is too conservative and misses real differences.
• If the larger group has the smaller variance, Student's test rejects the null hypothesis too often, producing more false positives than the stated 5%.

So Welch costs very little when variances are equal and protects you when they are not. This is the argument made by Delacre, Lakens and Leys (2017) in the International Review of Social Psychology for using Welch's test by default. It is also the default in R's t.test function.

Here is how much the choice can matter. With a group of 12 (mean 74.2, SD 4.0) and a group of 40 (mean 67.5, SD 14.0), Student's test gives t(50) = 1.63, p = .110, while Welch's gives t(49.99) = 2.68, p = .010. Same data, different conclusions, because the large group's large variance dominates the pooled estimate.

For paired data, compute each pair's difference and analyse those differences.

• Paired t-test: tests whether the mean difference is zero. It assumes the differences are roughly normally distributed (the original measurements need not be). With moderate or large numbers of pairs it is fairly robust.
• Wilcoxon signed-rank test: the rank-based alternative, for ordinal data or small samples where the differences are clearly non-normal or have outliers. It assumes the distribution of differences is roughly symmetric.
• Sign test: uses only the direction of each difference. It makes the fewest assumptions but has the least power.

• Independence comes from the design, not from a test. Repeated measurements on the same people, or students clustered within classes, violate it and need a different analysis (for example, a paired test or a multilevel model).
• Normality is best judged by looking: a histogram, box plot or normal Q–Q plot of each group (or of the paired differences). Formal tests such as Shapiro–Wilk flag trivial departures in large samples and miss important ones in small samples.
• Outliers: investigate them before deciding anything. Remove a value only if it is a genuine error, and report what you did and why.

The t-tests are fairly robust to moderate non-normality, particularly with similar group sizes and samples of a few dozen or more. Severe skew, heavy tails or extreme outliers in small samples are the cases where a rank-based test is worth switching to.

Whichever test you use, report enough for a reader to judge the result:

• descriptive statistics for each group (n, mean and SD, or median and interquartile range for ranked data),
• the test by name (say "Welch's t-test", not just "t-test"),
• the test statistic, degrees of freedom and exact p value,
• an effect size, such as Cohen's d for t-tests or the rank-biserial correlation for Mann–Whitney,
• a confidence interval for the difference where possible.

Keep significance and importance separate. A small p value says the data are unlikely under the null hypothesis; it does not say the difference is large or meaningful. The effect size and confidence interval do that job. And a non-significant result is not evidence that the groups are the same.

Running a t-test for every pair of three or more groups inflates the chance of a false positive. Use a one-way ANOVA (Welch's ANOVA if variances differ) or the Kruskal–Wallis test for ranks, followed by planned comparisons or post-hoc tests with an appropriate correction. When reporting, follow your citation style for statistical notation, and present the comparison in a figure that shows the data, as described in how to make a publication-ready figure.