Two-Sample (Student's) t-Test

In plain language

The two-sample t-test checks whether the average value of some measurement (like total healthcare cost or days in hospital) is genuinely different between two groups of patients, or whether the observed gap could plausibly be explained by random chance. It computes how many "standard errors" the observed gap is from zero, then looks that up in a table to get a p-value. The most useful output is not the p-value but the confidence interval on the mean difference — a range that tells you both the direction and the plausible size of the gap in real-world units like dollars or days. It assumes both groups have similar spread in their data; when that assumption looks shaky, the closely related Welch t-test is a safer choice.

What the test does and why it matters

The two-sample (Student's) t-test asks one question: is the observed difference between two group means large enough relative to the variability in the data to be incompatible with a null hypothesis of zero difference? It quantifies the answer as a t statistic — the ratio of the mean difference to the standard error of that difference — and computes a p-value by comparing that statistic to a t-distribution whose width is governed by the degrees of freedom. In health economics and outcomes research, however, the p-value is rarely the deliverable that matters. Payers, HTA bodies, and clinical decision-makers need the mean difference itself and its 95% confidence interval: a number on a clinically interpretable scale (e.g., "the treated group had $1,840 lower total costs per patient, 95% CI $620 to $3,060") rather than a binary significant/not-significant verdict.

The mechanics: pooled variance and why equal variance is the load-bearing assumption

Student's t-test assumes that both groups share the same population variance. Under this assumption, the two within-group sum-of-squares estimates are pooled into a single, more precise estimate of the common variance:

pooled_var = (SS_1 + SS_2) / (n_1 + n_2 - 2)

where SS_1 and SS_2 are the within-group sums of squared deviations from each group mean. The degrees of freedom follow directly: df = n_1 + n_2 - 2. The standard error of the mean difference is then:

SE = sqrt(pooled_var * (1/n_1 + 1/n_2))

and the t statistic is:

t = (mean_1 - mean_2) / SE

The CI on the mean difference is:

(mean_1 - mean_2) +/- t(df, alpha/2) SE

where t(df, alpha/2) is the critical value from the t-distribution at the chosen significance level (1.96 in large samples; wider in small samples — for df = 8, t = 2.306 at alpha = 0.05). This CI is the effect-size estimate that belongs in every HEOR table and report.

The equal-variance assumption problem: why Welch is the better default

The equal-variance (homoscedasticity) assumption is rarely verified and frequently violated. In RWE datasets, treated and control groups often differ not just in mean outcome but also in variance — sicker patients cluster in one arm, older patients in another, and these systematic differences affect the spread of cost and utilization distributions just as much as the mean. When variances differ, the pooled-variance formula is biased and the nominal type-I error rate is not maintained.

Welch's t-test relaxes the equal-variance assumption by computing a separate variance estimate for each group and adjusting the degrees of freedom downward using the Welch-Satterthwaite approximation. Simulation studies by Delacre, Lakens, and Leys (2017) demonstrate that Welch's t-test maintains the nominal type-I error rate across a broad range of variance ratios and sample sizes, while Student's (pooled) t-test inflates type-I error when variances differ and loses only a trivial amount of power when variances happen to be equal. The actionable rule: Welch's t-test should be the default for two-sample continuous comparisons in applied work. Student's t-test has a legitimate role when there is a theoretical reason to assume equal variances (e.g., randomized experiments with moderate sample sizes and a continuous outcome known to have stable variance) or when matching the historical standard for a pre-specified primary analysis.

Why Student's t-test persists despite Welch being statistically superior

Student's t-test was introduced in 1908 by William Sealy Gosset (publishing under the pseudonym "Student"), working at the Guinness brewery on the problem of making inferences from small samples. For over a century it was the dominant introductory test in statistics curricula, and the equal-variance assumption was treated as a "checking step" (via Levene's or Bartlett's test) rather than a known limitation. Many regulatory submissions and published clinical trials pre-specified Student's t-test for historical reasons. Understanding Student's t-test is therefore essential for interpreting legacy literature and for knowing exactly when the Welch modification is warranted.

The t statistic, degrees of freedom, and the t-distribution

The t statistic measures how many standard errors the observed mean difference is from zero. When the null hypothesis is true (zero population mean difference) and both normality and equal-variance assumptions hold, the t statistic follows a t-distribution with df = n_1 + n_2 - 2. The t-distribution has heavier tails than the normal distribution — especially at small df — which correctly reflects the additional uncertainty introduced by estimating the variance from the data rather than knowing it exactly. As df increases (larger samples), the t-distribution approaches the standard normal. For practical purposes, at df > 30 the difference is small; at df > 100 it is negligible.

The Central Limit Theorem and robustness at large n

Student's t-test does not require the raw data to be normally distributed — it requires the sampling distribution of the mean difference to be approximately normal. The Central Limit Theorem guarantees this for large enough samples regardless of the underlying data distribution. At n > 30 per group, the test is generally robust to moderate skewness; at n > 100 per group, it is robust to most departures from normality short of extreme heavy tails or structural zeros. This has a counterintuitive implication emphasized by Fagerland (2012): in large RWE datasets (n = 10,000 or more per group), the normality assumption is essentially irrelevant, yet the t-test p-value is almost always significant for trivially small differences. In this regime, the CI on the mean difference — and whether the difference is clinically or economically meaningful — is the only quantity worth reporting.

RWE realities: scale, confounding, and the limits of unadjusted two-group comparisons

At claims-scale n, the two-sample t-test is mechanically robust (CLT protects the p-value) but inferentially limited in three important ways.

First, at very large n, p-values become uninformative — a difference of $12 in mean total cost between two groups of 50,000 patients will be highly significant but is clinically and economically irrelevant. The CI is what matters, and in large samples the CI is often narrow enough to exclude any difference of practical importance.

Second, an unadjusted two-group t-test in observational data estimates a biased, confounded contrast. Treatment groups in RWE differ on age, comorbidities, prior utilization, and dozens of other covariates that independently affect the outcome. An unadjusted t-test on costs or healthcare utilization between a treated and an untreated group in claims data is a descriptive comparison — it describes the observed mean difference in this specific dataset but provides no causal inference about the treatment effect. Causal questions must route to propensity-score weighting, matching, g-methods, or regression adjustment before any t-test is applied. After propensity-score weighting or matching, a weighted t-test or a regression model in the matched sample is appropriate for the primary estimate.

Third, skewed outcomes — particularly total costs and healthcare utilization counts — violate the spirit of a mean-focused test when the distribution is so heavily right-skewed that the mean is a poor summary of typical patient experience. A generalized linear model with a log link and gamma or Tweedie variance function matches the distributional shape, targets the mean on the original dollar scale via back-transformation, and accommodates covariate adjustment in a single model. Reserve the t-test for approximately symmetric continuous outcomes or use it as a sensitivity check alongside a gamma GLM for cost outcomes.

Effect sizes and clinical significance: what HEOR needs beyond the p-value

Cohen's d is the standardized effect size for the t-test: d = (mean_1 - mean_2) / pooled_SD. By convention, d = 0.2 is small, 0.5 is medium, and 0.8 is large, though these benchmarks were calibrated in psychology and may not translate to healthcare. In HEOR, the unstandardized mean difference in original units (dollars of total cost, days of hospital stay, points on a quality-of-life scale) with its CI is more interpretable than Cohen's d for decision-making. Report both: the raw mean difference with CI for the clinical audience and Cohen's d for comparability with the broader literature.

Pros, cons, and trade-offs

Pros of the two-sample (Student's) t-test: - Produces a directly interpretable effect estimate: the mean difference and its CI on the original scale of the outcome. - Computationally trivial; universally implemented in Python (scipy), R, and SAS. - Well-understood power properties: closed-form sample size formulas exist for study planning. - Robust to non-normality at large n via the CLT. - Historical standard that facilitates comparison with legacy literature and satisfies pre-specified SAP language that mandates Student's t-test. - The pooled-variance estimate is slightly more efficient than Welch when equal variances truly hold.

Cons of the two-sample (Student's) t-test: - The equal-variance (pooled) assumption is frequently violated in RWE and can inflate type-I error; Welch's t-test should be preferred unless variances are known to be equal. - At large n, p-values are significant for trivially small differences; the CI must be reported to assess clinical or economic relevance. - Unadjusted application to confounded observational contrasts produces biased estimates; adjustment methods are required before t-test results are interpreted causally. - Heavily skewed outcomes (costs, utilization) violate the usefulness of a mean-focused test; gamma GLM or bootstrap mean-difference estimation is preferred for primary inference. - Cannot handle clustered or paired data; paired t-test or mixed models are required when observations are not independent.

When to use

The two-sample Student's t-test is appropriate when:

• Two independent groups are being compared on a continuous outcome and the equal-variance
• The pre-specified SAP or protocol explicitly calls for Student's t-test (e.g., a
• The outcome is approximately symmetric or the sample is large enough for CLT protection
• The analysis is a pre-post randomized comparison with moderate sample sizes and a continuous
• Baseline characteristic comparisons in a randomized trial (Table 1): t-tests on continuous
• Sensitivity analysis: running Student's t-test alongside Welch's and a nonparametric test

When NOT to use

The two-sample Student's t-test is not appropriate — and may be actively misleading — in the following situations:

• Paired or clustered data: when patients are matched, or when the data have a nested
• Heteroscedastic groups: when the two groups have substantially different variances
• Heavily skewed outcomes with small or moderate n: when the outcome is right-skewed (costs,
• As primary causal evidence in unmatched observational data: an unadjusted t-test on
• Ordinal or binary outcomes: for ordinal scores, use Mann-Whitney or an ordinal regression
• When the policy-relevant quantity is cost impact in a decision model: budget-impact and
• At large n without reporting the CI: at n > 5,000 per group, virtually any mean difference

Interpreting the output

In the worked example, five patients in a care management program (Group A) averaged 6.0 emergency department visits in the 12 months post-diagnosis; five patients in standard care (Group B) averaged 12.0 visits. The pooled variance is 10.0, the standard error of the mean difference is 2.0, the t statistic is −3.0 (absolute value 3.0) on df = 8, the two-sided p-value is approximately 0.017, and the 95% confidence interval on the mean difference is (−10.61, −1.39) ED visits.

(1) Formal interpretation. The observed mean difference of −6.0 visits is 3.0 standard errors below zero. Under the null hypothesis of no difference and the equal-variance assumption, a t statistic with absolute value ≥ 3.0 on df = 8 arises by chance in approximately 1.7% of samples. The 95% CI (−10.61, −1.39) is constructed so that if the study were repeated many times under identical conditions, approximately 95% of such intervals would contain the true mean difference — it does not mean there is a 95% probability that the true value lies in this specific interval. Excluding zero, the interval is consistent with a true mean reduction of between 1.4 and 10.6 visits.

(2) Practical interpretation. The data are consistent with Group A averaging roughly 6 fewer ED visits per patient than Group B, with plausible values ranging from about 1 to 11 fewer visits. Because this is an unadjusted observational comparison in a small study, the difference may reflect both program effects and differences in who enrolled. A statistically significant p-value indicates the gap is unlikely to be pure sampling noise, but it does not establish causation — a larger, adjusted study is needed to separate treatment effects from confounding and to narrow the CI enough to support a budget-impact estimate.

Worked example

Scenario

A health outcomes analyst is comparing total emergency department (ED) visits in the 12 months after diagnosis for two groups of patients: five who received a new care management program (Group A) and five who received standard care (Group B). The analyst wants to know whether the program is associated with fewer ED visits on average, and computes the two-sample Student's t-test by hand to understand every step — from the pooled variance to the t statistic and confidence interval.

Dataset

Total ED visits in the 12 months post-diagnosis (n=5 per group). Values are small integers chosen so that means, pooled variance, standard error, and t statistic all come out to clean exact values.

Steps

• Compute group means. Group A: mean_A = (2+4+6+8+10)/5 = 30/5 = 6.0 visits. Group B: mean_B = (8+10+12+14+16)/5 = 60/5 = 12.0 visits. Mean difference = mean_A - mean_B = 6.0 - 12.0 = -6.0 visits.

• Compute within-group sums of squares. For Group A, deviations from mean_A = 6: (2-6)^2 + (4-6)^2 + (6-6)^2 + (8-6)^2 + (10-6)^2 = 16 + 4 + 0 + 4 + 16 = 40. For Group B, deviations from mean_B = 12: (8-12)^2 + (10-12)^2 + (12-12)^2 + (14-12)^2 + (16-12)^2 = 16 + 4 + 0 + 4 + 16 = 40. SS_A = 40, SS_B = 40.

• Compute pooled variance. df = n_A + n_B - 2 = 5 + 5 - 2 = 8. pooled_var = (SS_A + SS_B) / df = (40 + 40) / 8 = 80 / 8 = 10.0.

• Compute standard error of the mean difference. SE = sqrt(pooled_var (1/n_A + 1/n_B)) = sqrt(10 (1/5 + 1/5)) = sqrt(10 2/5) = sqrt(10 0.4) = sqrt(4) = 2.0.

• Compute the t statistic. The absolute mean difference is 6.0, SE is 2.0, so the magnitude of t is 6.0 / 2.0 = 3.0, in the negative direction because mean_A < mean_B, giving t = -3.0. With df = 8, the two-sided p-value for absolute t = 3.0 is approximately 0.017.

• Compute the 95% confidence interval. The critical value for df = 8 at alpha = 0.025 is t_crit = 2.306. The margin is t_crit SE = 2.306 2.0 = 4.612. The CI is mean_diff +/- margin = -6.0 +/- 4.612, giving a lower bound of -10.612 and an upper bound of -1.388. The 95% CI on the mean difference is (-10.61, -1.39) ED visits.

• Interpret. The care management program is associated with 6.0 fewer ED visits on average (95% CI 1.4 to 10.6 fewer; p = 0.017). Because this is an unadjusted observational comparison, the difference may reflect confounding rather than a causal effect of the program. Before drawing conclusions, baseline covariate differences between the groups should be assessed and adjustment methods applied if imbalance is present.

Result

mean_A = 30/5 = 6.0 visits, mean_B = 60/5 = 12.0 visits, mean difference = -6.0 visits. SS_A = 16 + 4 + 0 + 4 + 16 = 40, SS_B = 16 + 4 + 0 + 4 + 16 = 40. pooled_var = (40 + 40) / 8 = 80/8 = 10.0. SE = sqrt(4) = 2.0. Absolute t = 6.0/2.0 = 3.0 (negative direction), df = 8, p approximately 0.017. Margin = 2.306 * 2.0 = 4.612. 95% CI on mean difference = (-10.61, -1.39) ED visits.

Runnable example

import math
from scipy import stats

# ── Motivating dataset: total ED visits post-diagnosis (n=5 per group) ──
group_a = [2, 4, 6, 8, 10]   # care management program; mean = 6.0
group_b = [8, 10, 12, 14, 16] # standard care; mean = 12.0

n_a, n_b = len(group_a), len(group_b)
mean_a = sum(group_a) / n_a   # 6.0
mean_b = sum(group_b) / n_b   # 12.0
mean_diff = mean_a - mean_b   # -6.0

# ── 1. Manual pooled-variance computation ──
ss_a = sum((x - mean_a) ** 2 for x in group_a)  # 40.0
ss_b = sum((x - mean_b) ** 2 for x in group_b)  # 40.0
df = n_a + n_b - 2                               # 8
pooled_var = (ss_a + ss_b) / df                  # 10.0
pooled_sd = math.sqrt(pooled_var)                # ~3.162
se = math.sqrt(pooled_var * (1 / n_a + 1 / n_b)) # 2.0
t_manual = mean_diff / se                        # -3.0
print(f"Manual:  pooled_var={pooled_var:.1f}, SE={se:.3f}, t={t_manual:.3f}, df={df}")

# ── 2. Student's t-test (equal_var=True = pooled) ──
t_student, p_student = stats.ttest_ind(group_a, group_b, equal_var=True)
print(f"\nStudent's t-test: t={t_student:.4f}, p={p_student:.4f}")

# ── 3. 95% CI on the mean difference ──
t_crit = stats.t.ppf(0.975, df=df)  # ~2.306 at df=8
ci_lo = mean_diff - t_crit * se
ci_hi = mean_diff + t_crit * se
print(f"Mean difference: {mean_diff:.1f} (95% CI {ci_lo:.2f} to {ci_hi:.2f})")

# ── 4. Cohen's d ──
cohens_d = mean_diff / pooled_sd
print(f"Cohen's d: {cohens_d:.3f}  (|0.2|=small, |0.5|=medium, |0.8|=large)")

# ── 5. Welch's t-test (equal_var=False) for comparison ──
t_welch, p_welch = stats.ttest_ind(group_a, group_b, equal_var=False)
print(f"\nWelch's t-test:  t={t_welch:.4f}, p={p_welch:.4f}  (preferred default)")
print("Note: when variances are equal (as here), Welch and Student give the same answer.")
print("When variances differ, Welch controls type-I error; Student's may not.")

# ── Motivating dataset ──
group_a <- c(2, 4, 6, 8, 10)   # care management; mean = 6
group_b <- c(8, 10, 12, 14, 16) # standard care; mean = 12

# ── 1. Student's t-test (equal variances assumed) ──
res_student <- t.test(group_a, group_b, var.equal = TRUE)
cat("=== Student's t-test (pooled) ===\n")
print(res_student)
cat(sprintf("Mean difference: %.1f\n", diff(res_student$estimate)))
cat(sprintf("95%% CI: [%.2f, %.2f]\n", res_student$conf.int[1], res_student$conf.int[2]))

# ── 2. Cohen's d ──
pooled_sd <- sqrt(((5-1)*var(group_a) + (5-1)*var(group_b)) / (5+5-2))
cohens_d  <- (mean(group_a) - mean(group_b)) / pooled_sd
cat(sprintf("Cohen's d: %.3f\n", cohens_d))

# ── 3. Welch's t-test (preferred default; var.equal = FALSE is R's default) ──
res_welch <- t.test(group_a, group_b, var.equal = FALSE)
cat("\n=== Welch's t-test (unequal variances; R default) ===\n")
print(res_welch)
cat("Practical rule: always use var.equal = FALSE (Welch) unless the SAP pre-specifies\n")
cat("Student's t-test or equal variances are theoretically justified.\n")