Exact and Penalized-Likelihood Methods for Sparse Data

In plain language

When a study table has a zero in one cell — say, no patients in the comparison group experienced the outcome — the standard formula for an odds ratio divides by zero and gives an infinite answer, which is meaningless. Exact methods (such as Fisher's exact test and exact logistic regression) compute the answer from the actual counts directly, without relying on any large-sample approximation, so they always produce a finite, valid result. A related tool called Firth penalized regression adds a small mathematical adjustment that pulls the estimate away from infinity, making it especially useful when several variables are in the model at once. Both approaches are needed in real-world evidence studies that track rare outcomes, such as a serious side effect that occurred in only a handful of patients.

Exact and penalized-likelihood methods for sparse data

are the small-sample toolkit for the situation that pervades real-world safety and subgroup work: a rare exposure crossed with a rare outcome, a stratified table with several zero cells, or a logistic/Cox model in which one covariate level perfectly predicts the outcome. In these regimes the ordinary maximum-likelihood (ML) estimator and the chi-square/Wald machinery that surround it are not merely imprecise - they are wrong in a structured way: the ML odds ratio is biased away from the null in finite samples, and under separation the ML estimate diverges to ±infinity with a Wald standard error that explodes, so the confidence interval and p-value become meaningless. Two distinct repairs exist. Exact methods (Fisher's exact test and its r x c generalization; exact conditional logistic regression) compute the inference from the permutation distribution of the sufficient statistic conditional on the margins, requiring no large-sample approximation. Penalized-likelihood methods (Firth's correction) add a Jeffreys-prior penalty to the score equation that removes the leading O(1/n) bias and always yields finite estimates and profile-likelihood intervals even under complete separation.

Core conceptual distinction

The two families answer the same question with different machinery and different estimands of convenience. (1) Exact conditional inference conditions on the observed margins (the row/column totals, or the nuisance parameters in conditional logistic regression) and enumerates the exact distribution of the test statistic; its p-values and confidence intervals are guaranteed to have at-or-below-nominal type-I error in any sample size, at the cost of conservatism (discreteness gaps) and heavy computation as tables grow. (2) Firth penalized likelihood keeps the unconditional model, shrinks the estimate toward the null by an amount that vanishes as n grows, and returns a finite, median-unbiased-ish odds ratio with a profile-penalized-likelihood interval. The estimand is the same conditional odds ratio you intended to report; the methods differ in how they handle the absence of asymptotic support. A third, often-overlooked option - adding a Bayesian/data-augmentation prior or weakly-informative penalization - generalizes Firth and is what Greenland recommends when sparse-data bias is suspected but separation is not literal. Crucially, none of these is a "different model": they are inference repairs for the same causal/ associational target, chosen because the asymptotics that justify Wald/score inference have failed.

Pros, cons, and trade-offs

- vs ordinary (asymptotic) ML logistic/Cox regression with Wald intervals: Exact and Firth methods give valid, finite inference where ML gives infinite estimates, undefined Wald SEs, and anticonservative p-values. Cost: exact inference can be markedly conservative (true coverage well above 95%, p-values inflated) and computationally explosive for large or many-covariate models; Firth is fast and well-calibrated but the penalty shrinks toward the null, so it is not ideal when you specifically need an unbiased estimate of a large true effect. Prefer ML only when every relevant cell is comfortably non-sparse (a common rule of thumb is >=8-10 events per parameter and no zero cells); otherwise prefer Firth as the default repair and reserve exact for regulatory tables where guaranteed type-I control is required. - vs Firth penalized likelihood (when choosing among the repairs): Exact conditional logistic is the gold standard for guaranteed coverage in a single small 2x2 or matched stratified table and is the standard for matched case-control designs; Firth scales to multivariable models, time-to-event (Firth-corrected Cox), and is the pragmatic choice when several covariates must be adjusted. Prefer exact for small, low-dimensional, regulator- facing contingency analyses; prefer Firth for adjusted multivariable sparse models and any model with a continuous covariate (where exact enumeration is intractable). - vs collapsing categories / dropping the sparse stratum / "+0.5 to every cell" (Haldane-Anscombe): Naive fixes discard information or introduce an arbitrary, sample-size-dependent bias. The ad-hoc 0.5 continuity correction is a crude special case of penalization and performs poorly in multivariable settings. Prefer Firth/exact over any hand-edited cell counts in a deliverable. - vs Bayesian logistic with a weakly-informative prior: Conceptually adjacent (Firth is a Jeffreys-prior MAP); a normal/Cauchy prior gives the analyst explicit control of shrinkage and full posterior intervals. Prefer the Bayesian route when you want to encode genuine prior information or report a posterior; prefer Firth when you want a frequentist, prior-free default that referees recognize.

When to use

Rare outcomes (e.g., adjudicated serious adverse events, rare cancers) crossed with a small or rare exposure group; subgroup/interaction analyses that thin the data into near-empty cells; matched case-control analyses (conditional logistic) with few discordant pairs; any logistic or Cox model where the software reports a "quasi- complete separation" / "did not converge" warning or returns an odds ratio in the thousands with an enormous SE; post-marketing safety signal tables with zero events in one arm. Diagnose first: tabulate every covariate against the outcome, count events per parameter, and look for empty or near-empty cells before fitting.

When NOT to use - and when it is actively misleading or dangerous

- Do not use exact methods as a fishing expedition for "significance." Their conservatism means a non-significant exact p-value with wide intervals is often the honest answer that the data cannot support a conclusion; reporting a Firth point estimate without its (wide) interval to manufacture precision is the dangerous failure mode. - Sparse-data bias is not a small-sample-only problem. Greenland's central warning: a large overall N with a rare outcome and many adjustment covariates can still be sparse, and asymptotic ML can be biased away from the null even when nothing looks small in the row totals. Trusting the converged ML odds ratio here is the trap. - Firth shrinks toward the null. If the true effect is genuinely large and well-supported, Firth will understate it; do not use it as a universal default when the data are actually rich. - Exact does not fix confounding, selection, or measurement error. A perfectly valid exact p-value on a confounded contingency table is precisely confident nonsense. These are inference repairs, not design repairs - they belong downstream of an active-comparator new-user design and confounding control, never as a substitute for them. - Separation can signal a real data problem, e.g., an exclusion criterion accidentally encoded as a covariate, or a post-baseline variable on the causal pathway. Investigate the separating variable clinically before penalizing it away.

Data-source operational depth

- Claims (FFS vs MA): Rare-event safety counts are exquisitely sensitive to person-time completeness. Medicare Advantage enrollees lack fee-for-service claims, so an arm built from MA-only person-time can show zero events not because none occurred but because they were never observed - a structural zero that exact/Firth methods will dutifully analyze as if it were real. Restrict to A/B/D (or commercial medical+pharmacy) enrollees and exclude MA-only spans before counting. Claims-lag and run-out also manufacture spurious zeros in the most recent calendar period; freeze a claims-maturity cutoff. Differential competing risks by exposure in an elderly claims cohort can empty the event cell in the frailer arm (they die of something else first) - a sparse cell that is really a competing-risk artifact, not a protective effect. - EHR: Encounter-driven capture means a "zero" outcome in a subgroup may reflect patients who left the system, not true absence of events; sparse cells are often missingness cells. Confirm denominators with explicit observation windows and consider linkage before declaring a structural zero. - Registry: Adjudicated, complete event capture makes registry zeros more trustworthy than claims zeros, but rare strata still need exact/Firth inference; weak pharmacy exposure capture means the exposure cell can be undercounted. - Linked claims-EHR-vital records: The completeness needed to distinguish a true zero from an unobserved zero, but linkage selects the linkable subset and can itself thin small strata - report the linkage denominator alongside the sparse table.

Worked claims example

Question: incidence of a rare adjudicated serious adverse event (e.g., agranulocytosis) in new initiators of a niche second-line antithyroid drug (STUDY) vs an active comparator (COMP) among adults, in a commercial + Medicare FFS database. (1) Cohort: active-comparator new-user design - >=365 days of continuous A/B/D (or commercial medical+pharmacy) enrollment with no fill of either drug in the lookback, exclude MA-only person-time so a zero event count cannot be an observation artifact, index_date = first qualifying fill, arm assigned from the dispensed NDC. (2) Outcome: first inpatient/ED claim with the qualifying dx code in the first-listed position within an on-treatment window (last days_supply end + 30-day grace). (3) Suppose the crude 2x2 is STUDY 3/412 vs COMP 0/903. The zero in the comparator arm produces complete separation: ordinary `PROC LOGISTIC` returns an infinite odds-ratio estimate with a "quasi-complete separation" warning and a Wald 95% CI of (0, infinity) - uninterpretable. (4) Repair: exact conditional logistic (`PROC LOGISTIC ... / exact`) gives a finite exact odds ratio with a guaranteed-coverage exact CI and an exact p-value; Firth (`PROC LOGISTIC ... firth`) gives a finite penalized OR with a profile-likelihood CI for the adjusted model that also includes age, sex, and baseline comorbidity count - covariates that ordinary ML could not estimate alongside the separating exposure. (5) Report both the point estimate and the (wide) interval, and state explicitly that the comparator arm contributed zero events: the honest conclusion is a signal worth following, not a precise effect size.

Interpreting the output

Consider the 2×2 table above: 3 events in 412 study-drug patients, 0 events in 903 comparator patients. Ordinary logistic regression fails (infinite OR, Wald CI 0 to ∞). Exact conditional logistic regression returns a finite exact OR ≈ 15.4 with a wide but bounded exact CI, and an exact p-value.

(1) Formal statistical interpretation. The exact OR ≈ 15.4 is a point estimate derived from the conditional distribution of the 2×2 table, holding the marginal totals fixed. The accompanying exact CI has guaranteed coverage at the nominal level regardless of sample size, unlike Wald intervals that rely on large-sample normal approximations. The width of the interval directly reflects the information content of the data: with only three events and a zero cell, the data are compatible with a wide range of true odds ratios. The exact p-value is the probability, under the null of no association, of observing a configuration at least as extreme as the one seen — it is not the probability that the drug is truly harmful.

(2) Practical interpretation for a decision-maker. The result is a pharmacovigilance signal, not a precise effect estimate. An OR of ≈ 15 sounds alarming, but the wide CI honestly communicates that the true effect could plausibly be much smaller. The appropriate response is a larger follow-up study to narrow the interval, not a formulary decision based on a three-event table. Report both the point estimate and the full interval so readers understand both the signal and the uncertainty.

Worked example

Scenario

A claims-based safety study asks whether a niche antithyroid drug (STUDY arm) is associated with a rare blood disorder called agranulocytosis compared with an active comparator drug (COMP arm). After building a new-user cohort with continuous enrollment confirmed throughout follow-up, analysts count adjudicated agranulocytosis events: 3 events among 412 STUDY initiators and 0 events among 903 COMP initiators. The goal is a valid odds ratio and confidence interval comparing the two arms.

Dataset

Adjudicated event counts by treatment arm — the 2x2 table an analyst would construct from the analytic dataset.

Steps

• Step 1 — Attempt the standard odds ratio formula: OR = (events_STUDY / non_events_STUDY) / (events_COMP / non_events_COMP) = (3 / 409) / (0 / 903). The denominator is 0 / 903 = 0, so the division is undefined — the standard formula returns infinity.

• Step 2 — Attempt a Wald confidence interval: the Wald method requires taking the natural log of each cell count. ln(0) is negative infinity, so the standard error is undefined and the confidence interval cannot be computed. Statistical software will issue a quasi-complete separation warning and may print OR = infinity with CI = (0, infinity).

• Step 3 — Apply Fisher's exact test (exact conditional inference): instead of using a formula, the exact method enumerates all possible ways to arrange 3 total events across 1,315 patients while keeping the row and column totals fixed. From that exact distribution it computes a finite exact conditional odds ratio and a guaranteed-coverage 95% confidence interval. The result is a finite number (approximately 15.4) with a wide but interpretable confidence interval, and an exact p-value.

• Step 4 — Interpret honestly: the confidence interval is wide because the data are genuinely sparse — only 3 events total. The honest conclusion is that the STUDY arm shows a signal worth monitoring, not a precisely measured effect size. Both the point estimate and the wide interval must be reported, along with the fact that the COMP arm had zero events.

Result

Standard ML odds ratio: undefined (division by zero, software returns infinity). Exact conditional odds ratio: approximately 15.4 (exact 95% CI is finite and guaranteed-coverage but wide, reflecting only 3 total events across 1,315 patients). The exact method gives a valid, reportable answer where the standard formula fails entirely.

Runnable example

import pandas as pd
import numpy as np
from scipy.stats import fisher_exact
from firthlogist import FirthLogisticRegression  # pip install firthlogist

# --- (a) Crude 2x2: exact conditional inference (no large-sample approximation) ---
tab = pd.crosstab(df["arm"], df["event"]).reindex(
    index=["STUDY", "COMP"], columns=[1, 0])           # rows = arm, cols = event yes/no
odds_ratio, p_exact = fisher_exact(tab.values, alternative="two-sided")
# For a sparse zero-cell table, prefer an exact CI implementation (e.g., scipy.stats.contingency
# odds_ratio(kind='conditional').confidence_interval) which returns finite, guaranteed-coverage bounds:
from scipy.stats.contingency import odds_ratio as exact_or
res = exact_or(tab.values, kind="conditional")
ci_low, ci_high = res.confidence_interval(confidence_level=0.95)
print(f"Exact conditional OR={res.statistic:.3f}  95% CI=({ci_low:.3f}, {ci_high:.3f})  p={p_exact:.4f}")

# --- (b) Adjusted model: Firth penalized logistic (finite under separation) ---
X = df[["age", "sex", "comorbid_n"]].copy()
X["study"] = (df["arm"] == "STUDY").astype(int)        # exposure indicator
y = df["event"].to_numpy()
fl = FirthLogisticRegression(skip_lrt=False)           # profile-penalized-likelihood inference
fl.fit(X.to_numpy(), y)
summary = pd.DataFrame({
    "term": list(X.columns),
    "coef": fl.coef_,
    "OR": np.exp(fl.coef_),
    "ci_low": np.exp(fl.ci_[:, 0]),                    # profile-likelihood CI, NOT Wald
    "ci_high": np.exp(fl.ci_[:, 1]),
    "pval": fl.pvals_,
})
print(summary)

library(exact2x2)
library(logistf)

# --- (a) Crude 2x2: exact conditional test + guaranteed-coverage CI ---
tab <- with(df, table(factor(arm, levels = c("STUDY", "COMP")),
                      factor(event, levels = c(1, 0))))   # rows = arm, cols = event yes/no
ex <- exact2x2(tab, tsmethod = "central")                 # central -> matches exact CI convention
print(ex)                                                 # exact OR, exact 95% CI, exact p-value

# --- (b) Adjusted model: Firth penalized logistic (finite under separation) ---
df$study <- as.integer(df$arm == "STUDY")                 # exposure indicator
fit <- logistf(event ~ study + age + sex + comorbid_n,
               data = df, pl = TRUE)                       # pl = profile penalized likelihood
summary(fit)                                              # coefficients, profile-likelihood CIs, p-values
exp(cbind(OR = coef(fit), confint(fit)))                  # odds ratios with PL confidence limits