Two-Sample (Student's) t-Test

What the test does and why it matters

The two-sample (Student's) t-test asks one question: is the observed difference between two group means large enough relative to the variability in the data to be incompatible with a null hypothesis of zero difference? It quantifies the answer as a t statistic — the ratio of the mean difference to the standard error of that difference — and computes a p-value by comparing that statistic to a t-distribution whose width is governed by the degrees of freedom. In health economics and outcomes research, however, the p-value is rarely the deliverable that matters. Payers, HTA bodies, and clinical decision-makers need the mean difference itself and its 95% confidence interval: a number on a clinically interpretable scale (e.g., "the treated group had $1,840 lower total costs per patient, 95% CI $620 to $3,060") rather than a binary significant/not-significant verdict.

The mechanics: pooled variance and why equal variance is the load-bearing assumption

Student's t-test assumes that both groups share the same population variance. Under this assumption, the two within-group sum-of-squares estimates are pooled into a single, more precise estimate of the common variance:

pooled_var = (SS_1 + SS_2) / (n_1 + n_2 - 2)

where SS_1 and SS_2 are the within-group sums of squared deviations from each group mean. The degrees of freedom follow directly: df = n_1 + n_2 - 2. The standard error of the mean difference is then:

SE = sqrt(pooled_var * (1/n_1 + 1/n_2))

and the t statistic is:

t = (mean_1 - mean_2) / SE

The CI on the mean difference is:

(mean_1 - mean_2) +/- t*(df, alpha/2) * SE

where t*(df, alpha/2) is the critical value from the t-distribution at the chosen significance level (1.96 in large samples; wider in small samples — for df = 8, t* = 2.306 at alpha = 0.05). This CI is the effect-size estimate that belongs in every HEOR table and report.

The equal-variance assumption problem: why Welch is the better default

The equal-variance (homoscedasticity) assumption is rarely verified and frequently violated. In RWE datasets, treated and control groups often differ not just in mean outcome but also in variance — sicker patients cluster in one arm, older patients in another, and these systematic differences affect the spread of cost and utilization distributions just as much as the mean. When variances differ, the pooled-variance formula is biased and the nominal type-I error rate is not maintained.

Welch's t-test relaxes the equal-variance assumption by computing a separate variance estimate for each group and adjusting the degrees of freedom downward using the Welch-Satterthwaite approximation. Simulation studies by Delacre, Lakens, and Leys (2017) demonstrate that Welch's t-test maintains the nominal type-I error rate across a broad range of variance ratios and sample sizes, while Student's (pooled) t-test inflates type-I error when variances differ and loses only a trivial amount of power when variances happen to be equal. The actionable rule: Welch's t-test should be the default for two-sample continuous comparisons in applied work. Student's t-test has a legitimate role when there is a theoretical reason to assume equal variances (e.g., randomized experiments with moderate sample sizes and a continuous outcome known to have stable variance) or when matching the historical standard for a pre-specified primary analysis.

Why Student's t-test persists despite Welch being statistically superior

Student's t-test was introduced in 1908 by William Sealy Gosset (publishing under the pseudonym "Student"), working at the Guinness brewery on the problem of making inferences from small samples. For over a century it was the dominant introductory test in statistics curricula, and the equal-variance assumption was treated as a "checking step" (via Levene's or Bartlett's test) rather than a known limitation. Many regulatory submissions and published clinical trials pre-specified Student's t-test for historical reasons. Understanding Student's t-test is therefore essential for interpreting legacy literature and for knowing exactly when the Welch modification is warranted.

The t statistic, degrees of freedom, and the t-distribution

The t statistic measures how many standard errors the observed mean difference is from zero. When the null hypothesis is true (zero population mean difference) and both normality and equal-variance assumptions hold, the t statistic follows a t-distribution with df = n_1 + n_2 - 2. The t-distribution has heavier tails than the normal distribution — especially at small df — which correctly reflects the additional uncertainty introduced by estimating the variance from the data rather than knowing it exactly. As df increases (larger samples), the t-distribution approaches the standard normal. For practical purposes, at df > 30 the difference is small; at df > 100 it is negligible.

The Central Limit Theorem and robustness at large n

Student's t-test does not require the raw data to be normally distributed — it requires the sampling distribution of the mean difference to be approximately normal. The Central Limit Theorem guarantees this for large enough samples regardless of the underlying data distribution. At n > 30 per group, the test is generally robust to moderate skewness; at n > 100 per group, it is robust to most departures from normality short of extreme heavy tails or structural zeros. This has a counterintuitive implication emphasized by Fagerland (2012): in large RWE datasets (n = 10,000 or more per group), the normality assumption is essentially irrelevant, yet the t-test p-value is almost always significant for trivially small differences. In this regime, the CI on the mean difference — and whether the difference is clinically or economically meaningful — is the only quantity worth reporting.

RWE realities: scale, confounding, and the limits of unadjusted two-group comparisons

At claims-scale n, the two-sample t-test is mechanically robust (CLT protects the p-value) but inferentially limited in three important ways.

First, at very large n, p-values become uninformative — a difference of $12 in mean total cost between two groups of 50,000 patients will be highly significant but is clinically and economically irrelevant. The CI is what matters, and in large samples the CI is often narrow enough to exclude any difference of practical importance.

Second, an unadjusted two-group t-test in observational data estimates a biased, confounded contrast. Treatment groups in RWE differ on age, comorbidities, prior utilization, and dozens of other covariates that independently affect the outcome. An unadjusted t-test on costs or healthcare utilization between a treated and an untreated group in claims data is a descriptive comparison — it describes the observed mean difference in this specific dataset but provides no causal inference about the treatment effect. Causal questions must route to propensity-score weighting, matching, g-methods, or regression adjustment before any t-test is applied. After propensity-score weighting or matching, a weighted t-test or a regression model in the matched sample is appropriate for the primary estimate.

Third, skewed outcomes — particularly total costs and healthcare utilization counts — violate the spirit of a mean-focused test when the distribution is so heavily right-skewed that the mean is a poor summary of typical patient experience. A generalized linear model with a log link and gamma or Tweedie variance function matches the distributional shape, targets the mean on the original dollar scale via back-transformation, and accommodates covariate adjustment in a single model. Reserve the t-test for approximately symmetric continuous outcomes or use it as a sensitivity check alongside a gamma GLM for cost outcomes.

Effect sizes and clinical significance: what HEOR needs beyond the p-value

Cohen's d is the standardized effect size for the t-test: d = (mean_1 - mean_2) / pooled_SD. By convention, d = 0.2 is small, 0.5 is medium, and 0.8 is large, though these benchmarks were calibrated in psychology and may not translate to healthcare. In HEOR, the unstandardized mean difference in original units (dollars of total cost, days of hospital stay, points on a quality-of-life scale) with its CI is more interpretable than Cohen's d for decision-making. Report both: the raw mean difference with CI for the clinical audience and Cohen's d for comparability with the broader literature.

Pros, cons, and trade-offs

Pros of the two-sample (Student's) t-test:

• Produces a directly interpretable effect estimate: the mean difference and its CI on the original scale of the outcome.
• Computationally trivial; universally implemented in Python (scipy), R, and SAS.
• Well-understood power properties: closed-form sample size formulas exist for study planning.
• Robust to non-normality at large n via the CLT.
• Historical standard that facilitates comparison with legacy literature and satisfies pre-specified SAP language that mandates Student's t-test.
• The pooled-variance estimate is slightly more efficient than Welch when equal variances truly hold.

Cons of the two-sample (Student's) t-test:

• The equal-variance (pooled) assumption is frequently violated in RWE and can inflate type-I error; Welch's t-test should be preferred unless variances are known to be equal.
• At large n, p-values are significant for trivially small differences; the CI must be reported to assess clinical or economic relevance.
• Unadjusted application to confounded observational contrasts produces biased estimates; adjustment methods are required before t-test results are interpreted causally.
• Heavily skewed outcomes (costs, utilization) violate the usefulness of a mean-focused test; gamma GLM or bootstrap mean-difference estimation is preferred for primary inference.
• Cannot handle clustered or paired data; paired t-test or mixed models are required when observations are not independent.

When NOT to use

The two-sample Student's t-test is not appropriate — and may be actively misleading — in the following situations:

• Paired or clustered data: when patients are matched, or when the data have a nested structure (e.g., multiple measurements per patient, patients nested in clinics), the independence assumption is violated. Use a paired t-test for 1:1 matched pairs or a mixed model for clustered data.
• Heteroscedastic groups: when the two groups have substantially different variances (a rule of thumb: variance ratio > 2 or < 0.5), use Welch's t-test, which does not assume equal variances and is the better default in essentially all applied settings.
• Heavily skewed outcomes with small or moderate n: when the outcome is right-skewed (costs, hospital LOS, utilization counts) and the sample is small (n < 50 per group), the CLT has not yet protected the sampling distribution of the mean. Use a nonparametric test, a transformation, or a GLM as the primary analysis; the t-test may serve as a sensitivity check.
• As primary causal evidence in unmatched observational data: an unadjusted t-test on confounded observational contrasts is a descriptive statistic, not a causal estimate. Confounding must be addressed via matching, weighting, or regression adjustment before a t-test is interpreted as estimating a treatment effect.
• Ordinal or binary outcomes: for ordinal scores, use Mann-Whitney or an ordinal regression model; for binary outcomes, use chi-square, Fisher's exact, or logistic regression.
• When the policy-relevant quantity is cost impact in a decision model: budget-impact and cost-effectiveness models require mean cost estimates that are internally consistent with the model's variance structure; gamma GLM with log link is the standard approach for mean cost inference in HEOR.
• At large n without reporting the CI: at n > 5,000 per group, virtually any mean difference will be statistically significant; reporting only the p-value from a t-test is misleading because it conveys no information about whether the difference is clinically or economically meaningful.

Interpreting the output

In the worked example, five patients in a care management program (Group A) averaged 6.0 emergency department visits in the 12 months post-diagnosis; five patients in standard care (Group B) averaged 12.0 visits. The pooled variance is 10.0, the standard error of the mean difference is 2.0, the t statistic is −3.0 (absolute value 3.0) on df = 8, the two-sided p-value is approximately 0.017, and the 95% confidence interval on the mean difference is (−10.61, −1.39) ED visits.

(1) Formal interpretation. The observed mean difference of −6.0 visits is 3.0 standard errors below zero. Under the null hypothesis of no difference and the equal-variance assumption, a t statistic with absolute value ≥ 3.0 on df = 8 arises by chance in approximately 1.7% of samples. The 95% CI (−10.61, −1.39) is constructed so that if the study were repeated many times under identical conditions, approximately 95% of such intervals would contain the true mean difference — it does not mean there is a 95% probability that the true value lies in this specific interval. Excluding zero, the interval is consistent with a true mean reduction of between 1.4 and 10.6 visits.

(2) Practical interpretation. The data are consistent with Group A averaging roughly 6 fewer ED visits per patient than Group B, with plausible values ranging from about 1 to 11 fewer visits. Because this is an unadjusted observational comparison in a small study, the difference may reflect both program effects and differences in who enrolled. A statistically significant p-value indicates the gap is unlikely to be pure sampling noise, but it does not establish causation — a larger, adjusted study is needed to separate treatment effects from confounding and to narrow the CI enough to support a budget-impact estimate.

Decision diagram

flowchart TD
  Start["Two independent groups<br/>Continuous outcome<br/>Goal: compare means"] --> EqVar{Are group variances<br/>approximately equal?}
  EqVar -->|"Yes — or SAP pre-specifies<br/>Student's t-test"| Student["Student's t-test<br/>(pooled variance)<br/>df = n1+n2-2"]
  EqVar -->|"No — or unknown<br/>(the common case)"| Welch["Welch's t-test<br/>(separate variances)<br/>df = Welch-Satterthwaite"]
  Student --> Output["Report: mean difference<br/>95% CI on mean difference<br/>Cohen's d<br/>(p-value secondary)"]
  Welch --> Output
  Output --> Causal{Is this an<br/>adjusted causal<br/>comparison?}
  Causal -->|"No — unadjusted<br/>observational data"| Warn["CAUTION: unadjusted t-test<br/>on confounded data is<br/>descriptive only — confounders<br/>must be addressed first"]
  Causal -->|"Yes — after matching,<br/>weighting, or in an RCT"| Conclude["Interpret CI as<br/>causal estimate of<br/>mean treatment effect"]

Worked example

Scenario

A health outcomes analyst is comparing total emergency department (ED) visits in the 12 months after diagnosis for two groups of patients: five who received a new care management program (Group A) and five who received standard care (Group B). The analyst wants to know whether the program is associated with fewer ED visits on average, and computes the two-sample Student's t-test by hand to understand every step — from the pooled variance to the t statistic and confidence interval.

Dataset

Total ED visits in the 12 months post-diagnosis (n=5 per group). Values are small integers chosen so that means, pooled variance, standard error, and t statistic all come out to clean exact values.

Steps

Result

mean_A = 30/5 = 6.0 visits, mean_B = 60/5 = 12.0 visits, mean difference = -6.0 visits. SS_A = 16 + 4 + 0 + 4 + 16 = 40, SS_B = 16 + 4 + 0 + 4 + 16 = 40. pooled_var = (40 + 40) / 8 = 80/8 = 10.0. SE = sqrt(4) = 2.0. Absolute t = 6.0/2.0 = 3.0 (negative direction), df = 8, p approximately 0.017. Margin = 2.306 * 2.0 = 4.612. 95% CI on mean difference = (-10.61, -1.39) ED visits.

Trade-offs

Runnable example

import math
from scipy import stats

# ── Motivating dataset: total ED visits post-diagnosis (n=5 per group) ──
group_a = [2, 4, 6, 8, 10]   # care management program; mean = 6.0
group_b = [8, 10, 12, 14, 16] # standard care; mean = 12.0

n_a, n_b = len(group_a), len(group_b)
mean_a = sum(group_a) / n_a   # 6.0
mean_b = sum(group_b) / n_b   # 12.0
mean_diff = mean_a - mean_b   # -6.0

# ── 1. Manual pooled-variance computation ──
ss_a = sum((x - mean_a) ** 2 for x in group_a)  # 40.0
ss_b = sum((x - mean_b) ** 2 for x in group_b)  # 40.0
df = n_a + n_b - 2                               # 8
pooled_var = (ss_a + ss_b) / df                  # 10.0
pooled_sd = math.sqrt(pooled_var)                # ~3.162
se = math.sqrt(pooled_var * (1 / n_a + 1 / n_b)) # 2.0
t_manual = mean_diff / se                        # -3.0
print(f"Manual:  pooled_var={pooled_var:.1f}, SE={se:.3f}, t={t_manual:.3f}, df={df}")

# ── 2. Student's t-test (equal_var=True = pooled) ──
t_student, p_student = stats.ttest_ind(group_a, group_b, equal_var=True)
print(f"\nStudent's t-test: t={t_student:.4f}, p={p_student:.4f}")

# ── 3. 95% CI on the mean difference ──
t_crit = stats.t.ppf(0.975, df=df)  # ~2.306 at df=8
ci_lo = mean_diff - t_crit * se
ci_hi = mean_diff + t_crit * se
print(f"Mean difference: {mean_diff:.1f} (95% CI {ci_lo:.2f} to {ci_hi:.2f})")

# ── 4. Cohen's d ──
cohens_d = mean_diff / pooled_sd
print(f"Cohen's d: {cohens_d:.3f}  (|0.2|=small, |0.5|=medium, |0.8|=large)")

# ── 5. Welch's t-test (equal_var=False) for comparison ──
t_welch, p_welch = stats.ttest_ind(group_a, group_b, equal_var=False)
print(f"\nWelch's t-test:  t={t_welch:.4f}, p={p_welch:.4f}  (preferred default)")
print("Note: when variances are equal (as here), Welch and Student give the same answer.")
print("When variances differ, Welch controls type-I error; Student's may not.")

# ── Motivating dataset ──
group_a <- c(2, 4, 6, 8, 10)   # care management; mean = 6
group_b <- c(8, 10, 12, 14, 16) # standard care; mean = 12

# ── 1. Student's t-test (equal variances assumed) ──
res_student <- t.test(group_a, group_b, var.equal = TRUE)
cat("=== Student's t-test (pooled) ===\n")
print(res_student)
cat(sprintf("Mean difference: %.1f\n", diff(res_student$estimate)))
cat(sprintf("95%% CI: [%.2f, %.2f]\n", res_student$conf.int[1], res_student$conf.int[2]))

# ── 2. Cohen's d ──
pooled_sd <- sqrt(((5-1)*var(group_a) + (5-1)*var(group_b)) / (5+5-2))
cohens_d  <- (mean(group_a) - mean(group_b)) / pooled_sd
cat(sprintf("Cohen's d: %.3f\n", cohens_d))

# ── 3. Welch's t-test (preferred default; var.equal = FALSE is R's default) ──
res_welch <- t.test(group_a, group_b, var.equal = FALSE)
cat("\n=== Welch's t-test (unequal variances; R default) ===\n")
print(res_welch)
cat("Practical rule: always use var.equal = FALSE (Welch) unless the SAP pre-specifies\n")
cat("Student's t-test or equal variances are theoretically justified.\n")

/* ── Create motivating dataset ── */
data work.ed_visits;
  input patient_id $ group $ ed_visits;
  datalines;
A1 A  2
A2 A  4
A3 A  6
A4 A  8
A5 A 10
B1 B  8
B2 B 10
B3 B 12
B4 B 14
B5 B 16
;
run;

/* ── PROC TTEST: outputs both Student (Pooled) and Welch (Satterthwaite) rows ── */
proc ttest data=work.ed_visits alpha=0.05;
  class group;      /* compare groups A vs B */
  var ed_visits;
  /* OUTPUT INTERPRETATION:
     - "Pooled" row     = Student's t-test (assumes equal variances)   -> t=-3.0, df=8
     - "Satterthwaite" row = Welch's t-test (unequal variances allowed) -> same here
     The two rows converge when variances are equal (as in this example).
     When variances differ, prefer Satterthwaite (Welch). */
run;

/* ── To compute Cohen's d manually in a DATA step ── */
proc means data=work.ed_visits noprint;
  class group;
  var ed_visits;
  output out=work.sumstats mean=mn var=vr n=nn;
run;

data work.cohens_d;
  set work.sumstats(where=(_type_=1));
  pooled_var + (nn-1)*vr;
  n_total + nn;
  mean_diff + mn;
  if _n_=2 then do;
    pooled_sd = sqrt(pooled_var / (n_total - 2));
    cohens_d  = mean_diff / pooled_sd;
    put "Pooled SD = " pooled_sd 6.4 " Cohen's d = " cohens_d 6.4;
  end;
run;

Citations

1. [1]Student. The probable error of a mean. Biometrika. 1908;6(1):1-25.
2. [2]Delacre M, Lakens D, Leys C. Why psychologists should by default use Welch's t-test instead of Student's t-test. International Review of Social Psychology. 2017;30(1):92-101.

1. [3]Fagerland MW. t-tests, non-parametric tests, and large studies — a paradox of statistical practice? BMC Medical Research Methodology. 2012;12:78.

1. [4]Altman DG, Bland JM. Statistics notes: The normal distribution. BMJ. 1995;310(6975):298.