Welch's t-Test (Unequal Variances)

What Welch's t-test is and why it exists

The classical Student's t-test for two independent groups rests on two assumptions: (1) the sampling distribution of the mean is approximately normal (protected by the Central Limit Theorem at large n), and (2) the two groups have equal population variances. The second assumption is far more restrictive in practice than the first, and it is routinely violated in observational healthcare data where treatment groups differ in case-mix, disease severity, and cost dispersion. Welch's t-test, proposed by B. L. Welch in 1947, relaxes the equal-variance assumption entirely by estimating a separate sample variance for each group and plugging those separate estimates into both the test statistic and an approximated degrees-of-freedom formula. The result is a test that is valid regardless of whether the group variances match.

The Welch test statistic is:

t = (x̄₁ − x̄₂) / sqrt(s₁²/n₁ + s₂²/n₂)

where x̄ᵢ, sᵢ², and nᵢ are the sample mean, sample variance, and sample size for group i. The denominator is the standard error of the mean difference using separate (unpooled) variance estimates — compare this with Student's t, which pools the two variances into a single estimate under the equal-variance assumption.

The Welch-Satterthwaite degrees of freedom approximation

Because the Welch statistic is not exactly t-distributed, Welch (1947) and Satterthwaite (1946) independently derived an approximation for the effective degrees of freedom (df*):

df* = (s₁²/n₁ + s₂²/n₂)² / [(s₁²/n₁)²/(n₁−1) + (s₂²/n₂)²/(n₂−1)]

The value of df* falls between min(n₁, n₂) − 1 (the most conservative possible df) and n₁ + n₂ − 2 (the pooled df used by Student's t-test). When the two groups have equal variances and equal sample sizes, df* collapses to n₁ + n₂ − 2, exactly matching Student's t-test. When one group has a much larger variance or a much smaller sample, df* is pulled toward the smaller group's df, which widens the critical region appropriately. In practice, df* is rarely a round integer; software rounds it or uses a continuous t-distribution lookup. The key intuition: the test self-calibrates its conservatism based on the observed variance disparity — when one group drives nearly all the sampling uncertainty, the effective df reflects that group alone.

Why "test for equal variances first, then choose" is statistically harmful

A widely taught but statistically suboptimal workflow is: (1) Run Levene's test or Bartlett's test for equality of variances. (2) If p < 0.05, use Welch's t-test; otherwise, use Student's t-test.

Delacre, Lakens, and Leys (2017) demonstrated through simulation that this conditional strategy degrades type-I error control relative to simply always using Welch. The mechanism is straightforward: the preliminary variance test itself has a type-I error rate (it will occasionally falsely conclude equal variances when they differ, or vice versa), and that error cascades into the subsequent test choice. The resulting procedure has a composite error rate that is neither of the two individual tests' rates. Moreover, at small n the variance test has low power and will frequently fail to detect genuine heteroscedasticity, routing the analyst to Student's t-test exactly when the Welch correction is most needed.

The practical rule — now implemented as the default in R, and requiring an explicit flag in Python — is: always use Welch's t-test for two independent-sample continuous comparisons. If there is an explicit substantive reason to assume equal variances (e.g., a controlled laboratory experiment with a very carefully matched design), Student's t-test is permissible, but this is rare in healthcare data analysis.

Software defaults: R makes Welch the default; Python requires a flag; SAS prints both

• R: `t.test(x, y)` uses Welch by default (`var.equal = FALSE`). Student's t-test requires `var.equal = TRUE`. This is the correct default.
• Python (scipy): `scipy.stats.ttest_ind(x, y)` uses Student's t-test by default (`equal_var = True`). Welch requires `equal_var = False`. Analysts who do not set this flag are running Student's t-test without realising it — a common error.
• SAS (PROC TTEST): Outputs both the Satterthwaite (Welch) row and the Pooled (Student) row simultaneously. The analyst must read the correct row; the Satterthwaite row is appropriate by default for observational two-group comparisons. The COCHRAN option provides an additional Cochran-Cox approximation for very extreme variance ratios.

RWE realities: unequal variances are the norm, not the exception

In observational healthcare data, the treated and comparator groups routinely differ in ways that produce heteroscedasticity:

• Cost dispersion: A biologics-treated cohort may have far greater variance in total healthcare costs than a generic-treated cohort — high responders have near-zero costs, non-responders accumulate catastrophic costs, and the SD for the biologic arm can be many times that of the comparator.
• Utilization counts: Emergency department visits or inpatient days in a frail-elderly population versus a healthier comparator will differ vastly in their variance.
• Lab values: Disease-active patients may show far more within-group variability in biomarkers than stable patients.

In all these settings, Student's t-test — by pooling the variances — implicitly under- weights the group with the larger variance and produces a test statistic whose null distribution is misspecified. Welch's correction directly addresses this. At very large n (> 10,000 per group), the practical difference between Welch and Student is small, but the cost of using Welch is zero, so there is no reason to deviate from it.

Large-n behavior and the reporting imperative

At large sample sizes typical of administrative claims databases (n = 50,000 per arm), the Welch t-test will reject the null for mean differences so small they are clinically irrelevant — a difference of $2 in annual drug costs, or 0.01 days in length of stay. This is not a failure of the test; the test is doing exactly what it is designed to do. The failure would be interpreting statistical significance as clinical or policy relevance. Best practice in large observational datasets:

• Report the mean difference with a 95% confidence interval and label both quantities.
• Report an effect size (Cohen's d = mean difference / pooled SD) alongside the p-value.
• Explicitly comment on whether the magnitude of the difference is clinically meaningful.
• Note that the Welch t-test cannot adjust for confounding — it is a descriptive comparison or a valid causal test only in a randomized or perfectly balanced observational design.

Descriptive-only status in confounded observational comparisons

The Welch t-test, like all two-sample tests, estimates a raw (unadjusted) mean difference. In an unbalanced observational cohort, this raw difference confounds the treatment effect with case-mix differences between the groups. A statistically significant Welch t-test in an observational setting does not constitute causal evidence; it is a descriptive finding that motivates further adjusted analyses. When adjustment is needed, the appropriate tool is regression (linear regression for continuous outcomes, with or without covariate adjustment, which subsumes and extends the two-sample mean comparison), propensity-score weighting, or matching — not the two-sample test. Report the Welch t-test result in Table 1 of an observational study as descriptive evidence of baseline imbalance, not as a causal estimate.

Pros, cons, and trade-offs

Pros:

• Does not assume equal variances; valid across the full range of heteroscedasticity scenarios encountered in healthcare data.
• Computationally trivial; no external dependencies; available in every statistical language.
• Produces an interpretable mean difference with confidence interval — the policy-relevant quantity for budget-impact models and clinical decision summaries.
• Protected by the Central Limit Theorem for large n: robust to non-normality of the raw data when n is adequate (generally ≥ 30 per group, often less).
• Under equal variances and equal sample sizes, gives results essentially identical to Student's t-test — no cost to using it as the default.
• Better type-I error control than the conditional "test variances first" workflow across all variance-ratio and sample-size combinations (Delacre et al. 2017).

Cons:

• Assumes independence between groups — not valid for paired or matched data (use paired t-test or Wilcoxon signed-rank instead).
• Like all t-tests, targets the mean. For heavily skewed outcomes (costs, utilization) with small-to-moderate n, extreme outliers can distort the mean and therefore the test's conclusion; gamma GLM, bootstrap mean estimation, or Winsorization may be preferable.
• Does not adjust for confounders. As a descriptive/unadjusted test, it cannot produce causal mean-difference estimates in observational data without additional methods.
• At small n (< 15 per group) with severely non-normal data, the CLT has not yet stabilized the sampling distribution and the test may have poor type-I error control; Mann-Whitney U or a permutation test may be safer.
• The Satterthwaite df approximation is just that — an approximation. At very small n with extreme variance ratios, it may not be perfectly calibrated, though it remains superior to the pooled df of Student's t-test.

Trade-offs:

• Welch vs Student: always prefer Welch unless you have an explicit, defensible reason to assume equal variances. The performance penalty for Welch when variances happen to be equal is negligible.
• Welch t-test vs Mann-Whitney U: Welch targets the mean difference and produces an interpretable estimate on the original scale; Mann-Whitney tests stochastic dominance (P(X > Y) = 0.5) and requires the Hodges-Lehmann estimator for a companion effect estimate. For skewed data at small-to-moderate n, Mann-Whitney may have better type-I error control; for large n, the CLT makes Welch robust. These tests answer different questions and their results can legitimately diverge.
• Welch t-test vs gamma GLM: For cost outcomes where the mean is the policy-relevant quantity, a gamma GLM with log link is the modern standard because it (a) respects the right-skewed distributional shape, (b) produces a mean ratio on the original dollar scale, and (c) naturally accommodates covariate adjustment. The Welch t-test on raw costs is a reasonable exploratory and sensitivity tool but not the recommended primary method for cost inference.

When NOT to use

• Paired or matched data: if the two observations come from the same patient (pre-post, matched pair, crossover) the paired t-test or Wilcoxon signed-rank test is required. Using Welch on paired data ignores the within-patient correlation and wastes power.
• Skewed outcomes at small n: for cost, utilization, or time-to-event data with fewer than ~30 observations per group, the CLT has not stabilized the mean's sampling distribution. A Mann-Whitney U test or nonparametric bootstrap is safer.
• As causal evidence in an unbalanced observational comparison: a significant Welch t-test between a treatment and comparator arm in an unmatched, unweighted observational cohort is a descriptive finding only. Confounders can explain the entire difference. Route to regression adjustment, propensity-score methods, or g-methods for causal inference.
• When the target estimand is not the mean: for ordinal outcomes, bounded scores, or when the median or a rank-based quantity is substantively meaningful, Mann-Whitney U (with the Hodges-Lehmann estimate) or ordinal regression is more appropriate.
• As a substitute for adjustment when confounding is uncontrolled: do not report a Welch t-test p-value as primary evidence of a treatment effect in observational data — readers and payers will interpret it as causal even if the accompanying text disclaims it.
• For cost data as the primary confirmatory analysis when mean costs matter for budget impact: the gamma GLM or two-part model is the modern standard for mean cost inference in HEOR because it better accounts for the distributional shape and can incorporate covariate adjustment in a single coherent model.

Interpreting the output

In the worked example, Group A (n = 3) has a mean of 4.0 ED visits with a variance of approximately 0.667, while Group B (n = 3) has a mean of 6.0 visits with a variance of approximately 3.333 — a variance ratio of approximately 5:1. The Welch standard error is 1.0, the Welch t statistic is −2.0, and the Welch-Satterthwaite degrees of freedom are approximately 4. The critical value for a two-sided test at df = 4, alpha = 0.05 is approximately 2.78. Because |t| = 2.0 < 2.78, the test does not reach statistical significance (p > 0.05).

(1) Formal interpretation. The Welch t statistic of −2.0 does not exceed the critical boundary at df ≈ 4, so the observed mean difference of −2.0 visits is consistent with the null hypothesis of no difference at alpha = 0.05. The 95% CI on the mean difference — wider than under equal-variance assumptions because the Satterthwaite approximation reduces effective degrees of freedom — includes zero. The non-significant result does not prove the groups have equal means; it indicates that the evidence against equality falls short of the pre-specified threshold, partly because small sample sizes and unequal variances reduce the effective degrees of freedom. Had the analyst used Student's pooled t-test, the pooled variance would blend two very different within-group variances, yielding a misleading denominator for the test statistic.

(2) Practical interpretation. The 2-visit difference in average utilization between groups cannot be distinguished from chance variation at n = 3 per group with these unequal variances. The key methodological lesson is that the approximately 5:1 variance ratio required Welch's correction — pooling variances as Student's t-test does when they differ this much produces a test statistic whose null distribution is wrong. In a larger study, the same directional difference might yield a statistically significant result, but the current data are too sparse for the test to be informative.

Decision diagram

flowchart TD
  Q["Two independent groups,<br/>continuous outcome?"] --> EV["Are variances equal?"]
  EV -->|"Do NOT test first —<br/>just use Welch"| Welch["Welch's t-test<br/>(equal_var=False in Python;<br/>default in R;<br/>Satterthwaite row in SAS)"]
  Welch --> Result["Output: t-stat, Satterthwaite df*,<br/>p-value, mean diff, 95% CI"]
  Result --> Report["Report: mean difference + CI<br/>+ Cohen's d + p-value"]
  Result --> Large["Large n (>10k)?"]
  Large -->|Yes| EffSize["Focus on Cohen's d and CI;<br/>p-value will always be significant"]
  Large -->|No| Skew["Skewed data at small n?"]
  Skew -->|Yes| MWU["Consider Mann-Whitney U<br/>as primary or sensitivity analysis"]
  Skew -->|No| Done["Report Welch result as primary"]
  Report --> Confounded["Observational comparison?"]
  Confounded -->|Yes| Causal["Note: Welch is DESCRIPTIVE only —<br/>route to regression/PS methods<br/>for causal inference"]
  Confounded -->|No| RCT["RCT or balanced design:<br/>Welch t-test is valid<br/>for causal mean difference"]

Worked example

Scenario

A health outcomes analyst is comparing the number of outpatient visits in the 12 months after starting treatment between 4 patients on a new medication (Group A) and 4 patients on standard care (Group B). The two groups happen to have very different spreads: Group A patients are clustered tightly around a low visit count, while Group B has one patient with a very high count pulling up both the mean and the spread. The analyst uses Welch's t-test (unpooled) rather than Student's t-test (pooled) because the group standard deviations look very different, and verifies the arithmetic by hand.

Dataset

Annual outpatient visit counts for 4 patients per group. Group A is tightly clustered; Group B has one high-utilizer. All counts are integers for easy arithmetic.

Steps

Result

Group A mean = 16/4 = 4 visits, Group B mean = 24/4 = 6 visits, mean difference = -2. Variance_A = 2/3 = 0.667, Variance_B = 10/3 = 3.333 (ratio approximately 5:1). Welch SE = sqrt(0.1667 + 0.8333) = sqrt(1.0) = 1.0. Welch t = -2.0 / 1.0 = -2.0. Satterthwaite df* is approximately 4. Critical t at df=4, alpha=0.05 (two-sided) = 2.78. |t| = 2.0 < 2.78, so the result is NOT statistically significant. Welch is more conservative than Student here (Welch df = 4 vs Student df = 6) because Group B drives most of the sampling uncertainty — exactly the behavior the correction is designed to produce.

Trade-offs

Runnable example

import math
from scipy import stats

# ── Motivating dataset: outpatient visits (n=4 per group) ──
group_a = [3, 4, 4, 5]   # mean=4, variance=0.667 (tightly clustered)
group_b = [4, 5, 7, 8]   # mean=6, variance=3.333 (more spread)

# ── 1. Welch t-test (REQUIRED: equal_var=False; Python default is Student's pooled) ──
t_welch, p_welch = stats.ttest_ind(group_a, group_b, equal_var=False)
mean_a = sum(group_a) / len(group_a)
mean_b = sum(group_b) / len(group_b)
mean_diff = mean_a - mean_b
print(f"Welch t-test:  t={t_welch:.4f}, p={p_welch:.4f}")
print(f"Mean A={mean_a:.3f}, Mean B={mean_b:.3f}, mean difference={mean_diff:.3f}")
print("NOTE: equal_var=False is required for Welch; omitting it gives Student's t-test.\n")

# ── 2. Manual 95% CI for the mean difference ──
n_a, n_b = len(group_a), len(group_b)
var_a = sum((x - mean_a) ** 2 for x in group_a) / (n_a - 1)
var_b = sum((x - mean_b) ** 2 for x in group_b) / (n_b - 1)
se = math.sqrt(var_a / n_a + var_b / n_b)
# Satterthwaite df
df_star = (var_a / n_a + var_b / n_b) ** 2 / (
    (var_a / n_a) ** 2 / (n_a - 1) + (var_b / n_b) ** 2 / (n_b - 1)
)
t_crit = stats.t.ppf(0.975, df=df_star)
ci_lower = mean_diff - t_crit * se
ci_upper = mean_diff + t_crit * se
print(f"Satterthwaite df*: {df_star:.4f}")
print(f"SE of mean difference: {se:.4f}")
print(f"95% CI for (A - B): [{ci_lower:.4f}, {ci_upper:.4f}]")

# ── 3. Effect size: Cohen's d (using pooled SD, standard convention) ──
pooled_sd = math.sqrt(((n_a - 1) * var_a + (n_b - 1) * var_b) / (n_a + n_b - 2))
cohens_d = mean_diff / pooled_sd
print(f"\nCohen's d (A - B): {cohens_d:.4f}")
print("Interpretation: |d| < 0.2 small, 0.2-0.5 medium, > 0.8 large (Cohen 1988).\n")

# ── 4. Contrast with Student's pooled t-test (DO NOT USE as default — shown for comparison) ──
t_student, p_student = stats.ttest_ind(group_a, group_b, equal_var=True)
print(f"Student's t-test (equal_var=True, pooled — NOT recommended as default):")
print(f"  t={t_student:.4f}, p={p_student:.4f}, df={n_a + n_b - 2} (fixed pooled df)")
print(f"Welch is more conservative here (df*={df_star:.2f} vs {n_a+n_b-2}) due to unequal variances.")

# ── Motivating dataset ──
group_a <- c(3, 4, 4, 5)   # mean = 4, var = 0.667
group_b <- c(4, 5, 7, 8)   # mean = 6, var = 3.333

# ── 1. Welch t-test (R default: var.equal = FALSE) ──
# No extra argument needed — R already uses Welch as the default.
welch <- t.test(group_a, group_b)   # var.equal = FALSE is the default
print(welch)
cat(sprintf("\nMean A = %.3f, Mean B = %.3f, difference = %.3f\n",
            mean(group_a), mean(group_b), mean(group_a) - mean(group_b)))
cat(sprintf("Satterthwaite df* = %.4f\n", welch$parameter))
cat(sprintf("95%% CI for (A - B): [%.4f, %.4f]\n",
            welch$conf.int[1], welch$conf.int[2]))

# ── 2. Student's pooled t-test for contrast (NOT the recommended default) ──
student <- t.test(group_a, group_b, var.equal = TRUE)
cat(sprintf("\nStudent's t-test (var.equal = TRUE, pooled — for comparison only):\n"))
cat(sprintf("  t = %.4f, df = %.1f, p = %.4f\n",
            student$statistic, student$parameter, student$p.value))
cat(sprintf("  df = %d (fixed pooled) vs Welch df* = %.2f (Satterthwaite)\n",
            length(group_a) + length(group_b) - 2, welch$parameter))

# ── 3. Cohen's d (pooled SD) ──
n_a <- length(group_a); n_b <- length(group_b)
pooled_sd <- sqrt(((n_a - 1) * var(group_a) + (n_b - 1) * var(group_b)) / (n_a + n_b - 2))
cohens_d  <- (mean(group_a) - mean(group_b)) / pooled_sd
cat(sprintf("\nCohen's d (A - B) = %.4f\n", cohens_d))

# ── 4. Note on the var.equal argument ──
# In R, always use t.test(x, y) — the default IS Welch. Only set var.equal = TRUE
# if you have an explicit substantive reason to assume equal variances (rare in HEOR).

/* ── Create motivating dataset ── */
data work.visits;
  input patient_id $ group $ visits;
  datalines;
A1 A 3
A2 A 4
A3 A 4
A4 A 5
B1 B 4
B2 B 5
B3 B 7
B4 B 8
;
run;

/* ── Welch t-test via PROC TTEST ──
   SAS prints BOTH rows automatically:
     "Pooled"         -> Student's equal-variance t-test (pooled df = n1+n2-2)
     "Satterthwaite"  -> Welch t-test (Welch-Satterthwaite adjusted df)
   Always use the SATTERTHWAITE row for observational two-group comparisons.
   The COCHRAN option adds the Cochran-Cox approximation (useful for extreme variance ratios).
*/
proc ttest data=work.visits cochran;
  class group;        /* variable identifying the two groups (A vs B)        */
  var visits;         /* continuous outcome variable                          */
  /* To extract the Satterthwaite row programmatically, add:
     ods output ttests = work.ttest_out;
     Then filter: where method = 'Satterthwaite';                            */
run;

/* ── Note on reading the output ──
   The "Equality of Variances" section prints a folded F-test.
   Its p-value should NOT be used to decide which t-test row to report.
   Best practice: always report the Satterthwaite (Welch) row.
   If variances are actually equal, Welch and Student give nearly identical results.
   If variances differ, Welch is the correct answer.                         */

/* ── ODS output extraction for programmatic downstream use ── */
ods output ttests = work.ttest_results;
proc ttest data=work.visits;
  class group;
  var visits;
run;
ods output close;

/* Extract Welch (Satterthwaite) result */
data work.welch_result;
  set work.ttest_results;
  where method = 'Satterthwaite';
  label method   = "Test type (Satterthwaite = Welch)"
        tvalue   = "t-statistic"
        df       = "Degrees of freedom (Welch-Satterthwaite)"
        probt    = "Two-tailed p-value";
run;
proc print data=work.welch_result noobs label; run;

Citations

1. [1]Welch BL. The generalization of 'Student's' problem when several different population variances are involved. Biometrika. 1947;34(1-2):28-35.
2. [2]Fagerland MW. t-tests, non-parametric tests, and large studies — a paradox of statistical practice? BMC Medical Research Methodology. 2012;12:78.

1. [3]Delacre M, Lakens D, Leys C. Why psychologists should by default use Welch's t-test instead of Student's t-test. International Review of Social Psychology. 2017;30(1):92-101.