Welch's t-Test (Unequal Variances)

In plain language

Welch's t-test compares the average values of a measurement between two independent groups — for example, average annual healthcare costs for patients on a new drug versus patients on the standard drug — and asks whether any observed difference is larger than you would expect from chance alone. Unlike the older Student's t-test, Welch's version does not require the two groups to have the same amount of spread (variance) in their data, which matters because treated and control patients in real-world studies almost always differ in how spread out their outcomes are. The test produces a p-value and a mean difference with a confidence interval — but it cannot account for differences in patient characteristics between the groups, so a significant result in an observational study is a description of the data, not proof that the treatment caused the difference.

What Welch's t-test is and why it exists

The classical Student's t-test for two independent groups rests on two assumptions: (1) the sampling distribution of the mean is approximately normal (protected by the Central Limit Theorem at large n), and (2) the two groups have equal population variances. The second assumption is far more restrictive in practice than the first, and it is routinely violated in observational healthcare data where treatment groups differ in case-mix, disease severity, and cost dispersion. Welch's t-test, proposed by B. L. Welch in 1947, relaxes the equal-variance assumption entirely by estimating a separate sample variance for each group and plugging those separate estimates into both the test statistic and an approximated degrees-of-freedom formula. The result is a test that is valid regardless of whether the group variances match.

The Welch test statistic is:

t = (x̄₁ − x̄₂) / sqrt(s₁²/n₁ + s₂²/n₂)

where x̄ᵢ, sᵢ², and nᵢ are the sample mean, sample variance, and sample size for group i. The denominator is the standard error of the mean difference using separate (unpooled) variance estimates — compare this with Student's t, which pools the two variances into a single estimate under the equal-variance assumption.

The Welch-Satterthwaite degrees of freedom approximation

Because the Welch statistic is not exactly t-distributed, Welch (1947) and Satterthwaite (1946) independently derived an approximation for the effective degrees of freedom (df*):

df* = (s₁²/n₁ + s₂²/n₂)² / [(s₁²/n₁)²/(n₁−1) + (s₂²/n₂)²/(n₂−1)]

The value of df falls between min(n₁, n₂) − 1 (the most conservative possible df) and n₁ + n₂ − 2 (the pooled df used by Student's t-test). When the two groups have equal variances and equal sample sizes, df collapses to n₁ + n₂ − 2, exactly matching Student's t-test. When one group has a much larger variance or a much smaller sample, df is pulled toward the smaller group's df, which widens the critical region appropriately. In practice, df is rarely a round integer; software rounds it or uses a continuous t-distribution lookup. The key intuition: the test self-calibrates its conservatism based on the observed variance disparity — when one group drives nearly all the sampling uncertainty, the effective df reflects that group alone.

Why "test for equal variances first, then choose" is statistically harmful

A widely taught but statistically suboptimal workflow is: (1) Run Levene's test or Bartlett's test for equality of variances. (2) If p < 0.05, use Welch's t-test; otherwise, use Student's t-test.

Delacre, Lakens, and Leys (2017) demonstrated through simulation that this conditional strategy degrades type-I error control relative to simply always using Welch. The mechanism is straightforward: the preliminary variance test itself has a type-I error rate (it will occasionally falsely conclude equal variances when they differ, or vice versa), and that error cascades into the subsequent test choice. The resulting procedure has a composite error rate that is neither of the two individual tests' rates. Moreover, at small n the variance test has low power and will frequently fail to detect genuine heteroscedasticity, routing the analyst to Student's t-test exactly when the Welch correction is most needed.

The practical rule — now implemented as the default in R, and requiring an explicit flag in Python — is: always use Welch's t-test for two independent-sample continuous comparisons. If there is an explicit substantive reason to assume equal variances (e.g., a controlled laboratory experiment with a very carefully matched design), Student's t-test is permissible, but this is rare in healthcare data analysis.

Software defaults: R makes Welch the default; Python requires a flag; SAS prints both

• R: `t.test(x, y)` uses Welch by default (`var.equal = FALSE`). Student's t-test
• Python (scipy): `scipy.stats.ttest_ind(x, y)` uses Student's t-test by default
• SAS (PROC TTEST): Outputs both the Satterthwaite (Welch) row and the Pooled

RWE realities: unequal variances are the norm, not the exception

In observational healthcare data, the treated and comparator groups routinely differ in ways that produce heteroscedasticity:

• Cost dispersion: A biologics-treated cohort may have far greater variance in total
• Utilization counts: Emergency department visits or inpatient days in a frail-elderly
• Lab values: Disease-active patients may show far more within-group variability in

In all these settings, Student's t-test — by pooling the variances — implicitly under- weights the group with the larger variance and produces a test statistic whose null distribution is misspecified. Welch's correction directly addresses this. At very large n (> 10,000 per group), the practical difference between Welch and Student is small, but the cost of using Welch is zero, so there is no reason to deviate from it.

Large-n behavior and the reporting imperative

At large sample sizes typical of administrative claims databases (n = 50,000 per arm), the Welch t-test will reject the null for mean differences so small they are clinically irrelevant — a difference of $2 in annual drug costs, or 0.01 days in length of stay. This is not a failure of the test; the test is doing exactly what it is designed to do. The failure would be interpreting statistical significance as clinical or policy relevance. Best practice in large observational datasets:

• Report the mean difference with a 95% confidence interval and label both quantities.
• Report an effect size (Cohen's d = mean difference / pooled SD) alongside the p-value.
• Explicitly comment on whether the magnitude of the difference is clinically meaningful.
• Note that the Welch t-test cannot adjust for confounding — it is a descriptive comparison

Descriptive-only status in confounded observational comparisons

The Welch t-test, like all two-sample tests, estimates a raw (unadjusted) mean difference. In an unbalanced observational cohort, this raw difference confounds the treatment effect with case-mix differences between the groups. A statistically significant Welch t-test in an observational setting does not constitute causal evidence; it is a descriptive finding that motivates further adjusted analyses. When adjustment is needed, the appropriate tool is regression (linear regression for continuous outcomes, with or without covariate adjustment, which subsumes and extends the two-sample mean comparison), propensity-score weighting, or matching — not the two-sample test. Report the Welch t-test result in Table 1 of an observational study as descriptive evidence of baseline imbalance, not as a causal estimate.

Pros, cons, and trade-offs

Pros: - Does not assume equal variances; valid across the full range of heteroscedasticity scenarios encountered in healthcare data. - Computationally trivial; no external dependencies; available in every statistical language. - Produces an interpretable mean difference with confidence interval — the policy-relevant quantity for budget-impact models and clinical decision summaries. - Protected by the Central Limit Theorem for large n: robust to non-normality of the raw data when n is adequate (generally ≥ 30 per group, often less). - Under equal variances and equal sample sizes, gives results essentially identical to Student's t-test — no cost to using it as the default. - Better type-I error control than the conditional "test variances first" workflow across all variance-ratio and sample-size combinations (Delacre et al. 2017).

Cons: - Assumes independence between groups — not valid for paired or matched data (use paired t-test or Wilcoxon signed-rank instead). - Like all t-tests, targets the mean. For heavily skewed outcomes (costs, utilization) with small-to-moderate n, extreme outliers can distort the mean and therefore the test's conclusion; gamma GLM, bootstrap mean estimation, or Winsorization may be preferable. - Does not adjust for confounders. As a descriptive/unadjusted test, it cannot produce causal mean-difference estimates in observational data without additional methods. - At small n (< 15 per group) with severely non-normal data, the CLT has not yet stabilized the sampling distribution and the test may have poor type-I error control; Mann-Whitney U or a permutation test may be safer. - The Satterthwaite df approximation is just that — an approximation. At very small n with extreme variance ratios, it may not be perfectly calibrated, though it remains superior to the pooled df of Student's t-test.

Trade-offs: - Welch vs Student: always prefer Welch unless you have an explicit, defensible reason to assume equal variances. The performance penalty for Welch when variances happen to be equal is negligible. - Welch t-test vs Mann-Whitney U: Welch targets the mean difference and produces an interpretable estimate on the original scale; Mann-Whitney tests stochastic dominance (P(X > Y) = 0.5) and requires the Hodges-Lehmann estimator for a companion effect estimate. For skewed data at small-to-moderate n, Mann-Whitney may have better type-I error control; for large n, the CLT makes Welch robust. These tests answer different questions and their results can legitimately diverge. - Welch t-test vs gamma GLM: For cost outcomes where the mean is the policy-relevant quantity, a gamma GLM with log link is the modern standard because it (a) respects the right-skewed distributional shape, (b) produces a mean ratio on the original dollar scale, and (c) naturally accommodates covariate adjustment. The Welch t-test on raw costs is a reasonable exploratory and sensitivity tool but not the recommended primary method for cost inference.

When to use

• Two independent groups on a continuous outcome and the mean difference is the target
• Continuous baseline comparisons in Table 1 of any study (RCT or observational) as a
• Sensitivity analysis alongside a gamma GLM primary analysis for cost outcomes at large n,
• Any situation where Student's t-test is being considered — simply use Welch instead.
• Lab values, quality-of-life scores, or other approximately-normal continuous endpoints
• When communicating results to non-statistical audiences who understand mean differences

When NOT to use

• Paired or matched data: if the two observations come from the same patient (pre-post,
• Skewed outcomes at small n: for cost, utilization, or time-to-event data with fewer
• As causal evidence in an unbalanced observational comparison: a significant Welch
• When the target estimand is not the mean: for ordinal outcomes, bounded scores, or
• As a substitute for adjustment when confounding is uncontrolled: do not report a Welch
• *For cost data as the primary confirmatory analysis when mean costs matter for budget

Interpreting the output

In the worked example, Group A (n = 3) has a mean of 4.0 ED visits with a variance of approximately 0.667, while Group B (n = 3) has a mean of 6.0 visits with a variance of approximately 3.333 — a variance ratio of approximately 5:1. The Welch standard error is 1.0, the Welch t statistic is −2.0, and the Welch-Satterthwaite degrees of freedom are approximately 4. The critical value for a two-sided test at df = 4, alpha = 0.05 is approximately 2.78. Because |t| = 2.0 < 2.78, the test does not reach statistical significance (p > 0.05).

(1) Formal interpretation. The Welch t statistic of −2.0 does not exceed the critical boundary at df ≈ 4, so the observed mean difference of −2.0 visits is consistent with the null hypothesis of no difference at alpha = 0.05. The 95% CI on the mean difference — wider than under equal-variance assumptions because the Satterthwaite approximation reduces effective degrees of freedom — includes zero. The non-significant result does not prove the groups have equal means; it indicates that the evidence against equality falls short of the pre-specified threshold, partly because small sample sizes and unequal variances reduce the effective degrees of freedom. Had the analyst used Student's pooled t-test, the pooled variance would blend two very different within-group variances, yielding a misleading denominator for the test statistic.

(2) Practical interpretation. The 2-visit difference in average utilization between groups cannot be distinguished from chance variation at n = 3 per group with these unequal variances. The key methodological lesson is that the approximately 5:1 variance ratio required Welch's correction — pooling variances as Student's t-test does when they differ this much produces a test statistic whose null distribution is wrong. In a larger study, the same directional difference might yield a statistically significant result, but the current data are too sparse for the test to be informative.

Worked example

Scenario

A health outcomes analyst is comparing the number of outpatient visits in the 12 months after starting treatment between 4 patients on a new medication (Group A) and 4 patients on standard care (Group B). The two groups happen to have very different spreads: Group A patients are clustered tightly around a low visit count, while Group B has one patient with a very high count pulling up both the mean and the spread. The analyst uses Welch's t-test (unpooled) rather than Student's t-test (pooled) because the group standard deviations look very different, and verifies the arithmetic by hand.

Dataset

Annual outpatient visit counts for 4 patients per group. Group A is tightly clustered; Group B has one high-utilizer. All counts are integers for easy arithmetic.

Steps

• Compute group means. Group A: (3+4+4+5)/4 = 16/4 = 4. Group B: (4+5+7+8)/4 = 24/4 = 6.

• Compute sample variances (dividing by n-1 = 3). Group A deviations from mean 4: (3-4)^2 = 1, (4-4)^2 = 0, (4-4)^2 = 0, (5-4)^2 = 1. Sum = 2. Variance_A = 2/3 = 0.667. Group B deviations from mean 6: (4-6)^2 = 4, (5-6)^2 = 1, (7-6)^2 = 1, (8-6)^2 = 4. Sum = 10. Variance_B = 10/3 = 3.333.

• Compute the standard error of the mean difference (unpooled, Welch formula). SE = sqrt(Variance_A/n_A + Variance_B/n_B) = sqrt(0.667/4 + 3.333/4) = sqrt(0.1667 + 0.8333) = sqrt(1.0) = 1.0.

• Compute the Welch t-statistic. t = (mean_A - mean_B) / SE = (4 - 6) / 1.0 = -2.0.

• The Satterthwaite degrees of freedom (df) formula uses each group's contribution to the SE. Numerator = (0.667/4 + 3.333/4)^2 = (0.1667 + 0.8333)^2 = (1.0)^2 = 1.0. Denominator = (0.1667)^2/(n_A-1) + (0.8333)^2/(n_B-1) = 0.02779/3 + 0.69439/3 = 0.00926 + 0.23146 = 0.24072. df = 1.0 / 0.24072 = 4.15 (approximately 4 degrees of freedom). Note that df* = 4.15 falls between the lower bound of 3 (each group has n-1 = 4-1 = 3 degrees of freedom) and the upper bound of 4+4-2 = 6, as expected.

• For a two-sided test at df = 4.15 (using df = 4 as the conservative approximation), the critical t-value at alpha = 0.05 is approximately 2.78. Our |t| = 2.0 < 2.78, so we do not reject the null hypothesis at the 5% level. The observed mean difference of 2 visits per year (Group B higher) is not statistically significant at this sample size.

• For comparison, Student's pooled t-test would pool variances: pooled_var = ((3-1)0.667 + (3-1)3.333) / (4+4-2) = (20.667 + 23.333) / 6 = (1.333 + 6.667) / 6 = 8.0/6 = 1.333. Pooled SE = sqrt(1.333/4 + 1.333/4) = sqrt(0.3333 + 0.3333) = sqrt(0.6667) = 0.8165. Student t = -2.0 / 0.8165 = -2.449. With df = 6, the critical value is 2.45, barely not significant. Welch rightly produces a more conservative result (df = 4 vs 6) because Group B's much higher variance should widen the critical region.

Result

Group A mean = 16/4 = 4 visits, Group B mean = 24/4 = 6 visits, mean difference = -2. Variance_A = 2/3 = 0.667, Variance_B = 10/3 = 3.333 (ratio approximately 5:1). Welch SE = sqrt(0.1667 + 0.8333) = sqrt(1.0) = 1.0. Welch t = -2.0 / 1.0 = -2.0. Satterthwaite df* is approximately 4. Critical t at df=4, alpha=0.05 (two-sided) = 2.78. |t| = 2.0 < 2.78, so the result is NOT statistically significant. Welch is more conservative than Student here (Welch df = 4 vs Student df = 6) because Group B drives most of the sampling uncertainty — exactly the behavior the correction is designed to produce.

Runnable example

import math
from scipy import stats

# ── Motivating dataset: outpatient visits (n=4 per group) ──
group_a = [3, 4, 4, 5]   # mean=4, variance=0.667 (tightly clustered)
group_b = [4, 5, 7, 8]   # mean=6, variance=3.333 (more spread)

# ── 1. Welch t-test (REQUIRED: equal_var=False; Python default is Student's pooled) ──
t_welch, p_welch = stats.ttest_ind(group_a, group_b, equal_var=False)
mean_a = sum(group_a) / len(group_a)
mean_b = sum(group_b) / len(group_b)
mean_diff = mean_a - mean_b
print(f"Welch t-test:  t={t_welch:.4f}, p={p_welch:.4f}")
print(f"Mean A={mean_a:.3f}, Mean B={mean_b:.3f}, mean difference={mean_diff:.3f}")
print("NOTE: equal_var=False is required for Welch; omitting it gives Student's t-test.\n")

# ── 2. Manual 95% CI for the mean difference ──
n_a, n_b = len(group_a), len(group_b)
var_a = sum((x - mean_a) ** 2 for x in group_a) / (n_a - 1)
var_b = sum((x - mean_b) ** 2 for x in group_b) / (n_b - 1)
se = math.sqrt(var_a / n_a + var_b / n_b)
# Satterthwaite df
df_star = (var_a / n_a + var_b / n_b) ** 2 / (
    (var_a / n_a) ** 2 / (n_a - 1) + (var_b / n_b) ** 2 / (n_b - 1)
)
t_crit = stats.t.ppf(0.975, df=df_star)
ci_lower = mean_diff - t_crit * se
ci_upper = mean_diff + t_crit * se
print(f"Satterthwaite df*: {df_star:.4f}")
print(f"SE of mean difference: {se:.4f}")
print(f"95% CI for (A - B): [{ci_lower:.4f}, {ci_upper:.4f}]")

# ── 3. Effect size: Cohen's d (using pooled SD, standard convention) ──
pooled_sd = math.sqrt(((n_a - 1) * var_a + (n_b - 1) * var_b) / (n_a + n_b - 2))
cohens_d = mean_diff / pooled_sd
print(f"\nCohen's d (A - B): {cohens_d:.4f}")
print("Interpretation: |d| < 0.2 small, 0.2-0.5 medium, > 0.8 large (Cohen 1988).\n")

# ── 4. Contrast with Student's pooled t-test (DO NOT USE as default — shown for comparison) ──
t_student, p_student = stats.ttest_ind(group_a, group_b, equal_var=True)
print(f"Student's t-test (equal_var=True, pooled — NOT recommended as default):")
print(f"  t={t_student:.4f}, p={p_student:.4f}, df={n_a + n_b - 2} (fixed pooled df)")
print(f"Welch is more conservative here (df*={df_star:.2f} vs {n_a+n_b-2}) due to unequal variances.")

# ── Motivating dataset ──
group_a <- c(3, 4, 4, 5)   # mean = 4, var = 0.667
group_b <- c(4, 5, 7, 8)   # mean = 6, var = 3.333

# ── 1. Welch t-test (R default: var.equal = FALSE) ──
# No extra argument needed — R already uses Welch as the default.
welch <- t.test(group_a, group_b)   # var.equal = FALSE is the default
print(welch)
cat(sprintf("\nMean A = %.3f, Mean B = %.3f, difference = %.3f\n",
            mean(group_a), mean(group_b), mean(group_a) - mean(group_b)))
cat(sprintf("Satterthwaite df* = %.4f\n", welch$parameter))
cat(sprintf("95%% CI for (A - B): [%.4f, %.4f]\n",
            welch$conf.int[1], welch$conf.int[2]))

# ── 2. Student's pooled t-test for contrast (NOT the recommended default) ──
student <- t.test(group_a, group_b, var.equal = TRUE)
cat(sprintf("\nStudent's t-test (var.equal = TRUE, pooled — for comparison only):\n"))
cat(sprintf("  t = %.4f, df = %.1f, p = %.4f\n",
            student$statistic, student$parameter, student$p.value))
cat(sprintf("  df = %d (fixed pooled) vs Welch df* = %.2f (Satterthwaite)\n",
            length(group_a) + length(group_b) - 2, welch$parameter))

# ── 3. Cohen's d (pooled SD) ──
n_a <- length(group_a); n_b <- length(group_b)
pooled_sd <- sqrt(((n_a - 1) * var(group_a) + (n_b - 1) * var(group_b)) / (n_a + n_b - 2))
cohens_d  <- (mean(group_a) - mean(group_b)) / pooled_sd
cat(sprintf("\nCohen's d (A - B) = %.4f\n", cohens_d))

# ── 4. Note on the var.equal argument ──
# In R, always use t.test(x, y) — the default IS Welch. Only set var.equal = TRUE
# if you have an explicit substantive reason to assume equal variances (rare in HEOR).