concept

Parametric and Nonparametric Tests

A fundamental choice in statistical hypothesis testing: parametric tests (t-test, ANOVA, chi-square) assume the data come from a distributional family and make inferences about that family's parameters, while nonparametric tests (Mann-Whitney, Kruskal-Wallis, Wilcoxon signed-rank, Fisher exact) replace raw values with ranks or permutation counts and make fewer distributional assumptions — the right choice depends on data type, sample size, the specific question being asked, and what effect estimate is actually needed for decision-making.

Inferential_Statisticsstatisticsprimitivefoundationshypothesis-testingt-testparametricnonparametricMann-Whitney

Methods reference only. Use primary source citations and local policy before applying this in a study protocol, regulatory submission, payer dossier, or clinical decision.

In plain language

Parametric tests (like the t-test) and nonparametric tests (like the Mann-Whitney test) are both tools for deciding whether a difference between two groups is likely to be real or just chance — but they work differently. A parametric test assumes the numbers follow a bell-curve shape and uses that assumption to build the comparison; a nonparametric test replaces the actual numbers with their rank order (first, second, third...) and tests whether the ranks are randomly mixed between groups. The rank approach is more robust to outliers and skewed data, but the trade-off is that it answers a slightly different question and produces effect estimates (like a median shift) that are harder to plug into a cost model. For most large real-world datasets with skewed outcomes like healthcare costs, a specialized regression model is often a better choice than either test.

What "parametric" and "nonparametric" actually mean

A parametric test assumes the data were drawn from a distributional family — most often Gaussian (normal) — and constructs a test statistic whose sampling distribution is derived from that assumption (a t-distribution, F-distribution, or chi-squared distribution under the null). The t-test does not require that your data are exactly normal; it requires that the sample mean follows a normal sampling distribution, which the Central Limit Theorem (CLT) guarantees for large enough n regardless of the underlying data distribution. This distinction matters enormously in large real-world datasets.

A nonparametric test replaces the raw observations with their ranks (or counts, or permutation counts) and derives the null distribution from the combinatorics of those ranks rather than from a parametric family. This makes it robust to heavy tails, outliers, and severely skewed distributions. But it also changes what is being tested. A common misunderstanding is that the Mann-Whitney U test "tests whether the medians are equal." This is only true under the restrictive assumption that the two distributions differ only in location. In general, the Mann-Whitney test a hypothesis of stochastic dominance: under the null, the probability that a randomly chosen observation from group A exceeds a randomly chosen observation from group B equals 0.5. That is a subtly different question from a difference in means or medians, and the direction of that difference deserves explicit acknowledgment in any analysis report.

The decision tree by data type and question

The canonical test-selection logic by situation:

Continuous outcome, two independent groups: start with Welch's t-test (does not assume equal
Continuous outcome, paired / pre-post measurements: paired t-test (parametric) vs Wilcoxon
Continuous outcome, three or more groups: one-way ANOVA (parametric) vs Kruskal-Wallis
Binary or categorical outcome, two or more groups: Pearson chi-square test when expected
Ordinal outcome: Mann-Whitney for two groups; Kruskal-Wallis for multiple groups; do not
Correlation: Pearson r (parametric, assumes bivariate normality, measures linear
Skewed continuous outcomes in HEOR (costs, utilization): neither a bare rank test nor a

The Fagerland paradox: when does normality matter least?

A striking result from Fagerland (2012) is that the t-test's normality assumption is most defensible — because the CLT protects the sampling distribution of the mean — precisely in the large samples where analysts most often worry about non-normality and reach for a nonparametric test. Conversely, at small samples (n < 15 per group) where the CLT has not yet kicked in, the t-test's normality assumption is most critical, yet small studies often show "borderline normal" on the plot and proceed with a t-test anyway. The practical implication: in a 50,000-patient claims dataset, a t-test on skewed total costs is likely valid for inference on means; in a 12-patient pilot study with a heavy-tailed outcome, a nonparametric test or a transformation is genuinely safer.

Welch's t-test vs Student's t-test

Student's t-test assumes equal variances in the two groups. Welch's t-test relaxes that assumption by using a separate variance estimate per group and adjusting the degrees of freedom (the Welch-Satterthwaite equation). Simulation studies consistently show that using Welch as the default — rather than applying a preliminary test for equal variances and then choosing — yields better type-I error control across a broad range of scenarios (Delacre, Lakens, and Leys 2017). The convention of running Levene's test first and then choosing Student vs Welch is statistically suboptimal because the preliminary test adds its own error. The actionable rule: always use Welch's t-test for two-sample continuous comparisons unless there is an explicit theoretical reason to assume equal variances.

Normality testing is counterproductive in most applied settings

It is common practice to run a Shapiro-Wilk test first and "confirm" normality before proceeding to a t-test. This practice has two opposite failure modes: at small n (< 30), the Shapiro-Wilk test has low power and will fail to detect non-normality that would genuinely violate the t-test assumption; at large n (> 500), it will reject normality for trivial departures — a slightly heavy tail or minor skewness — that the CLT renders irrelevant for inference on means. The recommended approach in applied work is to decide by inspection (histogram, Q-Q plot) combined with subject-matter knowledge about the outcome distribution (costs are always right-skewed; lab values are approximately normal; time-to-event is rarely normal) rather than relying on a formal normality test to make the parametric/nonparametric choice.

Effect estimates and confidence intervals: what HEOR decision-making needs

A bare p-value from a rank test rarely provides what health economics and outcomes research needs. The deliverable for payers, HTA bodies, and clinical decision-makers is an effect estimate with a confidence interval on a clinically meaningful scale:

For a t-test or ANOVA: the mean difference (with 95% CI) is directly interpretable.
For a Mann-Whitney test: the Hodges-Lehmann estimator (the median of all pairwise differences)
For a chi-square or Fisher test: the risk difference, relative risk, or odds ratio with CI is
For cost data: the target estimand for decision modelling is almost always the mean cost (not

Pros, cons, and trade-offs

Parametric tests (t-test, ANOVA, chi-square): - Pros: directly interpretable effect estimates (mean differences, proportions); well-understood power properties; straightforward extension to regression for covariate adjustment; exact CI formulas; computationally trivial and widely implemented. Robust to non-normality at large n (CLT). Welch's t-test is robust to variance heterogeneity. - Cons: at small n with severely non-normal data, the distributional assumption may not hold and type-I error can be inflated; the chi-square approximation breaks down with sparse cells. - When to prefer: large n, approximately symmetric continuous outcomes, binary/count outcomes with adequate cell sizes, when a mean difference is the target quantity.

Nonparametric tests (Mann-Whitney, Kruskal-Wallis, Wilcoxon signed-rank, Fisher exact): - Pros: valid under broad distributional assumptions; robust to outliers and heavy tails; exact in small samples (Fisher, Wilcoxon); well-suited when the rank ordering is intrinsically meaningful (ordinal scales, bounded scores). - Cons: lower power than parametric tests when the parametric assumption holds; the effect estimate (Hodges-Lehmann shift, rank-biserial correlation) is less intuitive than a mean difference; rank tests are not directly extensible to covariate adjustment (regression analogues exist but are less familiar). - When to prefer: small n with clearly non-normal data; ordinal outcomes; sensitivity checks alongside a parametric primary analysis; situations where outlier contamination is expected.

Transformations and GLMs as the modern alternative: For outcomes that are inherently non-normal — costs, utilization counts, event rates — transformations (log, square root) or GLMs (Poisson/negative-binomial for counts; gamma/Tweedie for costs; logistic for binary) are often superior to both naive parametric tests and rank tests because they (a) match the distributional shape, (b) target inference on the original scale via back-transformation or marginal effects, (c) accommodate covariate adjustment naturally, and (d) remain valid under confounded observational designs when combined with weighting or matching.

When to use

Use parametric and nonparametric tests in their appropriate domain:

Primary analyses of randomized contrasts (RCTs, or observational studies where balance has
Baseline characteristic tables (Table 1 in observational studies): chi-square for
Sensitivity analyses: running a nonparametric test alongside a parametric primary analysis
Ordinal or bounded outcomes: rank tests or ordinal regression rather than a t-test on
Small-n pilot studies (n < 30 per group): nonparametric tests or exact tests are safer

When NOT to use — and when these tests are actively misleading

As the primary analysis when confounding is uncontrolled: an unadjusted t-test or Mann-Whitney
For cost data when the target estimand is the mean: a Mann-Whitney test on costs tells you
Chi-square with expected counts below 5: the chi-square approximation is unreliable in
Any unadjusted two-group test as the final word on an observational comparison: even in a
Interpreting a Mann-Whitney p-value as a test of medians: unless the two distributions
Using Shapiro-Wilk as a gating decision for parametric vs nonparametric: the test is

Implementation note for all three languages

The canonical side-by-side demonstration below shows, for each data type, the parametric and nonparametric function calls together with the Welch correction and Fisher exact alternative. All implementations use the same six-observation motivating dataset from the beginner layer (skewed, with one outlier) to show concretely that the t-test and Mann-Whitney can reach different inferential conclusions at small n — and why neither answer is wrong, they are just answering slightly different questions.

Interpreting the output

In the worked example, six rehabilitation-protocol patients (Group A) and six standard-care patients (Group B) are compared on days of missed work after a procedure. Group A has mean 3.0 days but includes one outlier (7 days); Group B has mean approximately 5.33 days with little internal spread. The Welch t statistic is approximately −2.535, with p ≈ 0.038 (nominally significant at alpha = 0.05). The Mann-Whitney rank sums are R_A = 27 and R_B = 51, total 78 = 12 × 13 / 2, confirming the computation. The Mann-Whitney two-sided p-value is approximately 0.065 — not significant at alpha = 0.05.

(1) Formal interpretation. The Welch t-test rejects the null of equal means (p ≈ 0.038) because the observed mean difference (3.0 vs 5.33 days) is large relative to the combined standard error, including the outlier's contribution to Group A's variance. The Mann-Whitney test does not reject the null of stochastic equality (p ≈ 0.065), because the outlier in Group A (7 days) contributes only the single highest rank (rank 12) rather than pulling the test statistic proportionally as it does the mean. The two tests are answering different questions: the t-test targets mean equality; the Mann-Whitney targets stochastic dominance (P(X_A > X_B) = 0.5). Both are technically valid at this n; the divergence arises because the outlier exerts a large influence on the mean but a bounded influence on rank ordering.

(2) Practical interpretation. When a mean difference is the target estimand — for example, mean missed-work days needed for a budget-impact model — the Welch t-test is the appropriate primary analysis, and the outlier is informative signal, not noise to suppress. When the question is rank-based ("does Group A tend toward fewer missed days?"), the Mann-Whitney is appropriate and naturally down-weights the extreme observation. Because the two tests reach different conclusions here, the pre-specified primary estimand must determine which drives the inference — this is precisely why the analysis plan should declare the primary estimand before seeing the data.

Worked example

Scenario

A health outcomes analyst is comparing days of missed work in the week after a procedure for six patients receiving a new rehabilitation protocol (Group A) versus six standard-care patients (Group B). The data are small, skewed, and one patient in Group A had an unusually high value (7 days). The analyst runs both a Welch t-test and a Mann-Whitney U test to see whether the two approaches agree, and manually computes the rank sums to understand why the nonparametric test reaches a different conclusion.

Dataset

Days of missed work post-procedure (n=6 per group). Group A has one outlier (7 days). All 12 values are pooled and ranked together for the Mann-Whitney test.

patient_id	group	days_missed
A1	A	1
A2	A	2
A3	A	2
A4	A	3
A5	A	3
A6	A	7
B1	B	4
B2	B	5
B3	B	5
B4	B	6
B5	B	6
B6	B	6

Steps

Result

Group A mean = 18/6 = 3.0 days, Group B mean = 32/6 = 5.33 days. Rank sum A = 27, rank sum B = 51, total = 78 = 1213/2. U_A = 27 - 21 = 6, U_B = 51 - 21 = 30, U_A + U_B = 36 = 66. The t-test is sensitive to the outlier (7 days in A) and yields a significant mean difference; the Mann-Whitney test downweights the outlier to a single high rank and yields a borderline result. Neither test is wrong; they are answering slightly different questions. For a mean- difference estimate (what a budget-impact model needs), the t-test or a GLM is appropriate; for a rank-based comparison robust to the outlier, the Mann-Whitney is appropriate with the Hodges-Lehmann shift estimate.

Runnable example

python implementation

Side-by-side parametric and nonparametric two-sample tests using scipy.stats. Demonstrates Welch t-test vs Mann-Whitney U for continuous outcomes, chi-square vs Fisher exact for binary 2x2 tables, and the Hodges-Lehmann location estimate. Uses the...

import math
from scipy import stats

# ── Motivating dataset: days of missed work (n=6 per group, Group A has outlier at 7) ──
group_a = [1, 2, 2, 3, 3, 7]   # mean = 3.0, pulled up by outlier
group_b = [4, 5, 5, 6, 6, 6]   # mean = 5.33, tightly clustered high

# ── 1. Welch t-test (parametric; does not assume equal variances) ──
t_stat, t_pval = stats.ttest_ind(group_a, group_b, equal_var=False)
mean_diff = sum(group_a)/len(group_a) - sum(group_b)/len(group_b)
print(f"Welch t-test:  t={t_stat:.3f}, p={t_pval:.4f}")
print(f"Mean A={sum(group_a)/len(group_a):.3f}, Mean B={sum(group_b)/len(group_b):.3f}")
print(f"Mean difference (A - B) = {mean_diff:.3f}")

# ── 2. Mann-Whitney U test (nonparametric; tests stochastic dominance, NOT medians) ──
u_stat, mw_pval = stats.mannwhitneyu(group_a, group_b, alternative="two-sided")
print(f"\nMann-Whitney U: U={u_stat:.1f}, p={mw_pval:.4f}")
print("Note: MWU tests P(A > B) = 0.5, NOT equality of medians (unless shapes are identical).")

# ── 3. Hodges-Lehmann estimator: median of all pairwise differences (A_i - B_j) ──
pairwise_diffs = sorted(a - b for a in group_a for b in group_b)
n = len(pairwise_diffs)
if n % 2 == 1:
    hl_estimate = pairwise_diffs[n // 2]
else:
    hl_estimate = (pairwise_diffs[n // 2 - 1] + pairwise_diffs[n // 2]) / 2
print(f"Hodges-Lehmann location estimate (A - B): {hl_estimate:.1f}")
print("(This is the rank-based 'shift' estimate that pairs with the Mann-Whitney test.)")

# ── 4. Chi-square vs Fisher exact for a binary 2x2 table ──
# Example: events (e.g., readmission) in two groups of 50
# Observed: [[events_A, no_event_A], [events_B, no_event_B]]
table = [[8, 42], [18, 32]]   # Group A: 8/50 events; Group B: 18/50 events
chi2, chi2_p, dof, expected = stats.chi2_contingency(table, correction=False)
print(f"\nChi-square: chi2={chi2:.3f}, p={chi2_p:.4f} (expected min cell: {expected.min():.1f})")
# Fisher exact: always valid; prefer when any expected count < 5
_, fisher_p = stats.fisher_exact(table)
print(f"Fisher exact:  p={fisher_p:.4f}")
print("Rule: use Fisher exact when any expected cell count < 5.")

# ── 5. Paired Wilcoxon signed-rank test (pre-post on same patients) ──
pre  = [10, 12, 14, 11, 13, 15]
post = [ 8, 10, 12,  9, 11, 12]
w_stat, wsr_p = stats.wilcoxon(pre, post)
t_paired, tp_p = stats.ttest_rel(pre, post)
print(f"\nPaired t-test:  t={t_paired:.3f}, p={tp_p:.4f}")
print(f"Wilcoxon SR:    W={w_stat:.1f}, p={wsr_p:.4f}")

r implementation

Side-by-side parametric and nonparametric tests in base R. Demonstrates t.test (Welch) vs wilcox.test for continuous outcomes, chisq.test vs fisher.test for binary tables, and the Hodges-Lehmann estimate via wilcox.test(conf.int=TRUE). Paired t.test vs...

# ── Motivating dataset ──
group_a <- c(1, 2, 2, 3, 3, 7)   # mean = 3.0; outlier at 7
group_b <- c(4, 5, 5, 6, 6, 6)   # mean = 5.33

# ── 1. Welch t-test (var.equal = FALSE is the R default for two-sample) ──
t_res <- t.test(group_a, group_b, var.equal = FALSE)
cat("Welch t-test:\n")
print(t_res)
# Mean difference and CI
cat(sprintf("Mean A = %.3f, Mean B = %.3f, diff = %.3f\n",
            mean(group_a), mean(group_b), mean(group_a) - mean(group_b)))

# ── 2. Mann-Whitney / Wilcoxon rank-sum test ──
mw_res <- wilcox.test(group_a, group_b, exact = FALSE)
cat("\nMann-Whitney U test:\n")
print(mw_res)
cat("Note: wilcox.test tests stochastic dominance (P(A>B)=0.5), not median equality.\n")

# ── 3. Hodges-Lehmann location estimate with CI ──
hl_res <- wilcox.test(group_a, group_b, conf.int = TRUE, exact = FALSE)
cat(sprintf("\nHodges-Lehmann estimate: %.1f  95%% CI: [%.1f, %.1f]\n",
            hl_res$estimate, hl_res$conf.int[1], hl_res$conf.int[2]))

# ── 4. Chi-square vs Fisher exact (binary 2x2 table) ──
tab <- matrix(c(8, 42, 18, 32), nrow = 2, byrow = TRUE,
              dimnames = list(c("GroupA","GroupB"), c("Event","NoEvent")))
cat("\nChi-square test:\n")
print(chisq.test(tab, correct = FALSE))
cat("\nFisher exact test:\n")
print(fisher.test(tab))

# ── 5. Paired t-test vs Wilcoxon signed-rank ──
pre  <- c(10, 12, 14, 11, 13, 15)
post <- c( 8, 10, 12,  9, 11, 12)
cat("\nPaired t-test:\n")
print(t.test(pre, post, paired = TRUE))
cat("\nWilcoxon signed-rank test:\n")
print(wilcox.test(pre, post, paired = TRUE, exact = FALSE))