Equivalence and Non-Inferiority Testing

In plain language

Equivalence and non-inferiority testing answer a different question from the usual hypothesis test: instead of asking "is treatment A better than B?", they ask "is treatment A no worse than B by more than a clinically meaningful amount?" To do this, analysts pre-specify a margin — the largest difference that would still leave the new treatment acceptable — and then use a confidence interval to show that the true difference almost certainly falls within that margin. A critical warning: when a standard study simply "fails to find a difference" (p > 0.05), that is not the same as demonstrating equivalence, because an underpowered study can also fail to detect a large and harmful difference. These methods are required for drug approvals involving biosimilars, generic substitutions, and formulary switches where treatments are assumed to work similarly.

The fundamental error: "p > 0.05" does not establish equivalence

The most consequential misconception in clinical and real-world research is interpreting a non-significant superiority test as evidence of equivalence or non-inferiority. When a study reports "no significant difference" (p > 0.05), all that has been demonstrated is that the data are statistically compatible with the null hypothesis of zero difference. But those same data may also be compatible — given the trial's precision — with differences of 3, 5, or 10 percentage points that would be highly clinically meaningful. This is the absence of evidence fallacy: failing to find evidence of a difference is not evidence of the absence of a meaningful difference.

An underpowered, poorly executed, or too-brief study can easily return p > 0.05 even when the test drug is substantially inferior. Declaring equivalence from that result licenses a potentially harmful switch. Equivalence and non-inferiority (NI) testing require a fundamentally different inferential architecture: the analyst must pre-specify a margin — the largest clinically acceptable difference — before data collection, and then demonstrate formally, using a confidence interval, that the true difference almost certainly falls within that margin. The evidentiary burden is placed on ruling out unacceptable inferiority, not merely on failing to detect superiority.

TOST mechanics: two one-sided tests

The Two One-Sided Tests (TOST) procedure formalizes equivalence and NI testing. For equivalence testing (symmetric, two-sided):

• H₀₁: δ ≤ −Δ (test drug is worse than comparator by more than Δ)
• H₀₂: δ ≥ +Δ (test drug is better than comparator by more than Δ)
• Equivalence is declared when both null hypotheses are rejected at level α.

Rejecting both H₀₁ and H₀₂ is mathematically equivalent to showing that the two-sided (1 − 2α) confidence interval for δ lies entirely within the interval (−Δ, +Δ). At the conventional α = 0.05 level, this means the 90% CI must be contained within the equivalence margins. At α = 0.025 per one-sided test (the common regulatory standard), the two-sided 95% CI must fall within the margins.

For non-inferiority testing (one-sided):

• H₀: δ ≥ Δ (test drug is inferior by more than the NI margin)
• Hₐ: δ < Δ (test drug is non-inferior)
• NI is declared when the upper bound of the (1 − α) one-sided CI — equivalently, the

The CI interpretation is unambiguous: if the upper bound of the 95% CI for the risk difference (test minus comparator, positive = test is worse) is below Δ, NI is demonstrated. If the entire 95% CI lies below zero (the test drug has definitively lower risk than the comparator), superiority is also demonstrated. If the entire CI falls within (−Δ, +Δ), formal equivalence is established.

NI margin selection: the M1/M2 framework

Choosing the NI margin is the most consequential and contested step. Two frameworks converge:

1. Clinical relevance. The margin should represent the largest clinically acceptable difference — the amount by which the test treatment could be worse while remaining acceptable to patients and prescribers given its other attributes (cost, tolerability, route). This is a judgment requiring clinical expert input.

2. Preserving a fraction of active-comparator effect (M1/M2 logic). Codified in ICH E10 and FDA NI guidance, this approach proceeds in two steps. M1 is the effect of the active comparator over placebo, estimated from a meta-analysis of historical placebo- controlled trials. M2 is the fraction of M1 that the test drug must preserve to be clinically acceptable — typically 50% or more for serious conditions. The NI margin equals M2 expressed in absolute terms.

In plain terms: if the reference drug reduces stroke risk by 10 percentage points compared to placebo (M1 = 10 pp), and clinicians require the test drug to preserve at least half of that benefit (50% × 10 pp = 5 pp = M2), the NI margin is 5 pp. The test drug may be at most 5 pp worse on absolute stroke risk.

ITT and per-protocol analysis roles are reversed in NI trials

In superiority trials, intent-to-treat (ITT) analysis is the primary analysis. Protocol deviations — patients who switched drugs, stopped treatment, or were misclassified — dilute the observed treatment effect toward zero. Dilution toward zero is conservative for a superiority claim (makes rejection of the null harder), so ITT protects against false- positive superiority findings.

In NI trials, the logic reverses entirely. Non-compliance, dropouts, contamination, and protocol deviations also dilute effects toward zero — and in an NI trial, a difference near zero is what demonstrates NI. A sloppily executed trial therefore produces spurious NI results: the test drug appears "no different" from the comparator precisely because the trial was too poorly conducted to detect anything. ITT is no longer the conservative analysis in NI settings; it can be actively anti-conservative.

Per-protocol analysis, which restricts to patients with full adherence and no major protocol violations, becomes co-primary in NI trials. Both ITT and per-protocol analyses must independently show NI for the conclusion to be credible. If ITT shows NI but per-protocol does not, sloppy execution is driving the result, not the drug's actual performance.

Assay sensitivity

Assay sensitivity is the implicit assumption that the trial could have detected a meaningful difference if one existed. For the M1/M2 logic to be valid, the historical evidence for M1 must be credible, and the trial conditions must be sufficiently similar to those historical trials. If assay sensitivity is doubtful — for example, the trial enrolled a low-risk population where even a highly effective drug shows little absolute benefit — the NI demonstration is uninterpretable. A trial that cannot distinguish an effective drug from an inactive one cannot establish NI.

RWE applications

Biosimilar and formulary-switch comparisons. Claims data are used to compare a biosimilar's real-world effectiveness against the reference biologic. The NI framework formalizes the question: is the biosimilar noninferior by more than the pre-specified margin on outcomes such as hospitalization or disease flare? Without an explicit NI margin, an underpowered comparison that "shows no difference" (p > 0.05) risks falsely licensing a harmful switch.

Cost-minimization prerequisites. Cost-minimization analysis (CMA) licenses comparing only costs — but only after equivalence on outcomes has been formally demonstrated. The NI or equivalence analysis supplies that evidentiary prerequisite. A non-significant superiority result does not provide this license; it merely reflects insufficient power.

Target-trial NI emulations. When emulating a target NI trial in real-world claims or linked data, confounding toward the null is the dominant hazard — and it is NOT reassuring. Residual confounding by indication can make a genuinely inferior therapy appear non-inferior because the "test" therapy is given to systematically healthier patients. Active-comparator new-user designs, propensity-score weighting, and negative-control analyses are all essential. Unlike in superiority emulations, where confounding toward the null is conservative, in NI emulations it produces a false-positive result.

Pros, cons, and trade-offs

NI/equivalence testing vs. standard superiority testing: - Pros: explicit, pre-specified acceptability threshold; CI-based conclusion with direct clinical interpretation; satisfies regulatory requirements for drug approvals and biosimilar designations; protects against licensing inferior drugs that happen to show p > 0.05 in underpowered trials. - Cons: the margin must be pre-specified and clinically justified — post-hoc margin setting is outcome-driven analysis and invalid; sample sizes are typically larger than for an equivalent-power superiority test; assay sensitivity is an untestable assumption that must be argued from context. - When to prefer: whenever the goal is to demonstrate that a treatment is "good enough" relative to an established comparator, not definitively superior.

Equivalence (symmetric, two-sided) vs. non-inferiority (one-sided): - Equivalence requires the test drug to be neither much worse NOR much better (the entire CI within ±Δ); used for bioequivalence (generic vs. brand, where superior efficacy in one direction is also undesirable) and for formally symmetric comparisons. - NI requires only that the test drug not be worse by more than Δ; it allows the test drug to be arbitrarily better. Used for most biosimilar, formulary-switch, and alternative- formulation comparisons.

ITT vs. per-protocol in NI settings: - Pros of per-protocol as co-primary: gives a cleaner signal of the intrinsic drug effect, uncontaminated by non-adherence; is the more stringent (harder to show NI) analysis. - Cons of relying on per-protocol alone: selection bias from excluding non-adherent patients; the excluded patients may be informatively different from adherent patients. - Requirement: both must agree for an NI conclusion to be credible.

When to use

• When the scientific question is whether a new or alternative treatment is clinically
• When a pre-specified NI margin can be justified from clinical reasoning and the M1/M2
• For biosimilar vs. reference biologic comparisons, generic vs. brand comparisons, oral vs.
• As the evidentiary prerequisite for a cost-minimization analysis.
• In target-trial NI emulations for biosimilar uptake, formulary switches, or care-setting

When NOT to use — and when it is actively misleading

• After seeing a non-significant superiority p-value, without pre-specifying the margin.
• When assay sensitivity cannot be established. If the trial or RWE study is too small,
• In confounded RWE analyses without rigorous design. Confounding toward the null
• When the margin is set to whatever makes the study pass. A margin chosen to ensure
• To satisfy a superiority requirement. An NI demonstration does not substitute for a
• When per-protocol and ITT analyses disagree. Concordance is required; disagreement

Interpreting the output

In the worked example, 80 events occurred among 2,000 test-drug initiators (risk = 0.040) and 72 events occurred among 2,000 comparator initiators (risk = 0.036). The observed risk difference is 0.004 (0.4 percentage points, test minus comparator). The 95% CI, computed by normal approximation, spans approximately −0.8 to 1.6 percentage points. The pre-specified NI margin is 2.0 percentage points.

(1) Formal interpretation. The estimand is the marginal risk difference in 30-day stroke or TIA risk between test-drug and comparator initiators in the matched cohort. The NI null hypothesis is H₀: δ ≥ 2.0 percentage points (test drug inferior by at least the margin). The observed risk difference of 0.4 pp has a 95% CI upper bound of 1.6 pp. Because 1.6 < 2.0, we reject H₀ and declare non-inferiority at the α = 0.025 one-sided level (equivalently, using the upper bound of the two-sided 95% CI). This CI has the repeated-sampling interpretation: in 95% of identically designed studies, the interval constructed this way would contain the true risk difference. All values in this interval are below the NI margin, so the data provide no support for clinically unacceptable inferiority. The CI crosses zero (lower bound −0.8 pp), so superiority of the test drug is not established.

(2) Practical interpretation. The test drug is non-inferior to the comparator on 30-day stroke/TIA risk: the most pessimistic plausible result (1.6 pp excess risk) still falls within the pre-agreed acceptable range (up to 2.0 pp). A decision-maker can read this as: even in the worst statistical case supported by these data, the test drug causes at most 1.6 extra strokes or TIAs per 100 patients — and the clinical team agreed before seeing any data that 2.0 extra events per 100 would still be acceptable given other benefits (cost, tolerability, route). The result does not mean the drugs are identical; it means the data cannot rule out a difference as large as 1.6 pp. Whether 1.6 pp matters for a particular formulary decision is a clinical and policy question, not a statistical one.

Worked example

Scenario

A health-outcomes analyst emulates a non-inferiority target trial comparing a new oral anticoagulant (test drug) against an established one (comparator) on 30-day stroke or transient ischemic attack (TIA) among adults initiating therapy in commercial claims data. The pre-specified non-inferiority margin is 2.0 percentage points (absolute risk difference), set using M1/M2 logic: the reference drug reduces 30-day stroke risk by approximately 4 pp vs. placebo (M1), and clinicians require the test drug to preserve at least half of that benefit, giving M2 = 2.0 pp as the margin. Both arms have 2,000 matched, continuously-enrolled patients followed from index fill.

Dataset

Summary of the non-inferiority analysis. Each row gives one arm's 30-day event count, sample size, and observed stroke/TIA risk. Both arms were matched on age, sex, comorbidities, and prior anticoagulant use.

Steps

• Confirm the pre-specified NI margin: Δ = 2.0 percentage points = 0.020 in proportion units. This margin was fixed before data collection using the M1/M2 framework.

• Compute risk in each arm: risk in test arm = 80/2000 = 0.040 (4.0 percentage points over 30 days); risk in comparator arm = 72/2000 = 0.036 (3.6 percentage points).

• Compute the risk difference: event count difference = 80 - 72 = 8; risk difference = 8/2000 = 0.004 (0.4 percentage points, test drug minus comparator — positive means the test drug had slightly more events, so there is a small observed penalty).

• The SE of the risk difference by normal approximation is approximately sqrt( 0.0400.960/2000 + 0.0360.964/2000) ≈ 0.0060. The 95% CI is approximately (0.004 - 1.960.0060, 0.004 + 1.960.0060) ≈ (-0.008, 0.016) in proportions, or (-0.8, 1.6) percentage points.

• NI decision: the 95% CI upper bound is approximately 0.016 (1.6 pp). The pre-specified NI margin is 0.020 (2.0 pp). Because the upper bound of 1.6 pp is less than the margin of 2.0 pp, the data provide no statistical support for clinically unacceptable inferiority and non-inferiority is demonstrated.

• Note what NI does NOT establish: the CI lower bound of approximately -0.8 pp crosses zero, so superiority of the test drug (i.e., definitively fewer events) is not established by these data. The result is NI only, not superiority.

Result

risk in test arm = 80/2000 = 0.040; risk in comparator arm = 72/2000 = 0.036; risk difference = 8/2000 = 0.004 (0.4 percentage points); 95% CI ≈ (-0.8 to 1.6) percentage points; CI upper bound 1.6 pp is below the NI margin of 2.0 pp; non-inferiority is demonstrated. Superiority is not established because the CI lower bound crosses zero.

Runnable example

import numpy as np
from scipy import stats
from statsmodels.stats.weightstats import ttost_ind

# ── 1. NI test on a binary risk difference ──────────────────────────────────────
# Worked example: 80/2000 vs 72/2000; NI margin = 0.020 (2.0 percentage points)

def ni_risk_diff(events_test, n_test, events_comp, n_comp,
                 ni_margin, alpha=0.025):
    """Non-inferiority test for a risk difference via the CI approach.

    H0: delta >= ni_margin  (test drug inferior by at least the margin)
    Ha: delta <  ni_margin  (test drug non-inferior)

    NI demonstrated when upper bound of two-sided (1-2*alpha) CI < ni_margin,
    equivalently when the one-sided z-test p-value < alpha.
    """
    p1 = events_test / n_test    # test drug risk
    p2 = events_comp / n_comp    # comparator risk
    rd = p1 - p2                 # risk difference (positive = test has more events)
    se = np.sqrt(p1*(1-p1)/n_test + p2*(1-p2)/n_comp)

    # Upper bound of two-sided (1-2*alpha) CI — the NI decision boundary
    z_crit = stats.norm.ppf(1 - alpha)   # 1.96 for alpha=0.025
    ci_upper = rd + z_crit * se
    ci_lower = rd - z_crit * se

    # One-sided z for H0: delta = ni_margin
    z_ni = (ni_margin - rd) / se
    p_one_sided = stats.norm.sf(z_ni)

    ni_achieved = ci_upper < ni_margin

    return {
        "risk_test":      round(p1, 6),
        "risk_comp":      round(p2, 6),
        "risk_diff":      round(rd, 6),
        "se":             round(se, 6),
        "ci_bounds":      (round(ci_lower, 6), round(ci_upper, 6)),
        "ni_margin":      ni_margin,
        "z_stat":         round(z_ni, 4),
        "p_one_sided":    round(p_one_sided, 4),
        "ni_demonstrated": ni_achieved,
    }

result = ni_risk_diff(80, 2000, 72, 2000, ni_margin=0.020, alpha=0.025)
print("NI result:", result)
# risk_diff = 0.004 (0.4 pp); ci_bounds ≈ (-0.008, 0.016);
# ci_upper 0.016 < margin 0.020 → ni_demonstrated = True

# ── 2. Equivalence (TOST) on a continuous outcome ───────────────────────────────
# ttost_ind(x1, x2, low, upp) tests H0a: delta <= low AND H0b: delta >= upp.
# Equivalence (low < delta < upp) is declared when max(p_low, p_high) < alpha.
rng = np.random.default_rng(42)
x_test = rng.normal(loc=5.0, scale=2.0, size=200)
x_comp = rng.normal(loc=5.1, scale=2.0, size=200)
equiv_margin = 1.0  # raw-unit equivalence half-width

# ttost_ind returns a 3-tuple: (pvalue, (t1, pv1, df1), (t2, pv2, df2))
# Do NOT unpack as five scalars — that will fail or misassign values.
tost_result = ttost_ind(
    x_test, x_comp,
    low=-equiv_margin, upp=equiv_margin,
    usevar="unequal",   # Welch correction
)
tost_p = tost_result[0]          # overall TOST p-value (max of the two one-sided p-values)
_, (t_low, p_low, df_low) = tost_result[0], tost_result[1]   # lower-bound test
_, (t_high, p_high, df_high) = tost_result[0], tost_result[2]  # upper-bound test
print(f"\nTOST p-value (overall): {tost_p:.4f}")
print(f"  Lower H0 (delta <= -{equiv_margin}): t={t_low:.3f}, p={p_low:.4f}")
print(f"  Upper H0 (delta >= +{equiv_margin}): t={t_high:.3f}, p={p_high:.4f}")
print(f"Equivalence demonstrated (alpha=0.05): {tost_p < 0.05}")
# tost_p is the binding (larger) one-sided p-value; both sub-tests must reject for equivalence.

# ── 1. NI test on a binary risk difference (manual CI) ──────────────────────────
events_test <- 80L;  n_test <- 2000L
events_comp <- 72L;  n_comp <- 2000L
ni_margin   <- 0.020   # 2.0 percentage points

p1 <- events_test / n_test   # 0.040
p2 <- events_comp / n_comp   # 0.036
rd <- p1 - p2                # 0.004

se <- sqrt(p1*(1-p1)/n_test + p2*(1-p2)/n_comp)
z_crit <- qnorm(0.975)       # 1.96 for two-sided 95% CI (alpha = 0.025 per side)
ci_upper <- rd + z_crit * se
ci_lower <- rd - z_crit * se

cat(sprintf("Risk difference: %.4f (%.2f pp)\n",   rd,       rd * 100))
cat(sprintf("95%% CI: (%.4f, %.4f) = (%.2f, %.2f) pp\n",
            ci_lower, ci_upper, ci_lower * 100, ci_upper * 100))
cat(sprintf("NI margin: %.3f (%.1f pp)\n",         ni_margin, ni_margin * 100))
cat(sprintf("NI demonstrated: %s\n",
            ifelse(ci_upper < ni_margin, "YES", "NO")))
# ci_upper ≈ 0.016 (1.6 pp) < 0.020 (2.0 pp) → NI demonstrated

# ── 2. Equivalence TOST on a continuous outcome (TOSTER) ────────────────────────
library(TOSTER)
set.seed(42)
x_test <- rnorm(200, mean = 5.0, sd = 2.0)
x_comp <- rnorm(200, mean = 5.1, sd = 2.0)
equiv_margin <- 1.0   # raw-unit equivalence half-width

# tost(): two one-sided Welch t-tests; equivalence when p_TOST < 0.05
tost_res <- tost(
  x = x_test, y = x_comp,
  low_eqbound  = -equiv_margin,
  high_eqbound =  equiv_margin,
  eqbound_type = "raw",
  alpha        = 0.05,
  var.equal    = FALSE    # Welch correction
)
print(tost_res)
# tost_res$TOST$p.value gives the max (binding) one-sided p;
# equivalence demonstrated when this p < alpha = 0.05.

# ── 3. Risk-difference CI from a 2x2 table using epitools ───────────────────────
library(epitools)
tab <- matrix(
  c(events_test, n_test - events_test,
    events_comp, n_comp - events_comp),
  nrow = 2, byrow = TRUE,
  dimnames = list(c("test_drug","comparator"), c("event","no_event"))
)
rr_out <- riskratio(tab, method = "wald")
# rr_out$measure: row "Relative Risk" gives RR; "Attributable Risk" gives risk diff
# Compare the attributable risk (RD) upper CI to the NI margin
cat("\nRisk ratio and attributable risk from epitools:\n")
print(rr_out$measure)