Firth Penalized Regression

In plain language

Firth penalized regression is a statistical method that fixes a specific failure of ordinary logistic regression: when every single person with a rare exposure has the outcome (or none does), the standard model breaks down and spits out an impossibly huge effect estimate. Firth adds a small mathematical correction to the model that pulls the estimate back to a finite, usable number while keeping it as accurate as the thin data allow. The trade-off is honest: when the data are this sparse, the resulting confidence interval will still be very wide, because the data genuinely cannot pin down the true effect.

Firth penalized regression

modifies the likelihood of a logistic or Cox model by adding a penalty term equal to one-half the log-determinant of the Fisher information (the Jeffreys invariant prior). Maximizing this penalized log-likelihood shifts the score equations by a term that exactly cancels the leading O(1/n) bias of the ordinary maximum likelihood estimate (MLE). The practical consequence in real-world evidence (RWE) is twofold: (1) coefficients are bias-reduced in small or imbalanced samples, and (2) — crucially for pharmacoepidemiology — estimates remain finite and unique even under complete or quasi-complete separation, the situation where an exposure or covariate perfectly (or nearly perfectly) predicts the outcome and the ordinary MLE diverges to ±infinity with an infinite standard error. Firth was conceived as a bias-reduction device (Firth 1993); Heinze and Schemper (2002) recognized it as the principled fix for separation, which is the reason it appears constantly in rare-outcome comparative-safety analyses.

Core conceptual distinction

. Firth is a finite-sample / sparse-data correction, not an alternative estimand. It targets the same conditional log–odds ratio or log–hazard ratio as ordinary logistic/Cox regression; it does not change what is being estimated, only how it is estimated when the data are too thin for the asymptotic MLE to behave. Distinguish it from three neighbours that are often conflated with it. (1) Exact logistic regression conditions on sufficient statistics and gives exact small-sample inference, but is computationally intractable for the multi-covariate, high-dimensional confounder sets typical of claims studies — Firth is the scalable approximation that keeps continuous covariates and many terms. (2) Ridge/LASSO penalize the size of coefficients toward zero to manage variance/selection; Firth penalizes via the information matrix to remove bias and guarantee finiteness, and (with the appropriate profile-penalized-likelihood interval) it is invariant to predictor scaling in a way ridge is not. (3) Exact/Poisson or rare-event corrections (e.g., King–Zeng) address a related rare-events problem but with a different correction; Firth is the more general, model-internal solution. A subtle but important caveat: standard Firth biases the intercept (and therefore predicted probabilities) toward 1/2, so for absolute-risk prediction in rare outcomes use FLIC (Firth logistic regression with intercept correction) or FLAC; for the comparative log–odds ratio that drives most RWE, plain Firth is appropriate.

Pros, cons, and trade-offs

(specific and comparative). - vs ordinary (unpenalized) logistic/Cox MLE: Firth always returns a finite estimate and a usable confidence interval under separation, where the MLE returns ±infinity, an absurd odds ratio (e.g., OR = 4.7e8), or a model that silently fails to converge. Even without separation it has smaller mean-squared error at low events-per-variable (EPV). Cost: it is a shrinkage estimator, so in large, well-separated-from-the-boundary samples it adds a touch of bias toward the null relative to the (then-unbiased) MLE, and Wald intervals are unreliable — you must use penalized profile-likelihood intervals and penalized likelihood-ratio tests, not Wald z/SE. - vs exact logistic regression: Firth scales to realistic claims confounder sets and continuous covariates and is far faster; exact methods give guaranteed small-sample coverage but become infeasible beyond a handful of categorical predictors. - vs dropping/collapsing the offending variable or using a non-informative prior in a Bayesian model: Firth keeps the variable in the model and is reproducible and deterministic; the Bayesian data-augmentation prior of Greenland and Mansournia (a weakly informative prior on the log-OR) is an excellent, often preferable alternative when you want a transparently chosen prior or posterior interval, and is conceptually close to Firth. - vs ridge/LASSO penalization: Firth does not select variables and does not require cross-validating a tuning parameter; choose ridge/LASSO when the goal is high-dimensional prediction or selection, Firth when the goal is a (near-)unbiased, finite effect estimate under sparsity.

When to use

. Use Firth when the analytic model has low EPV (a common rule of thumb is EPV < 10, with risk continuing below ~20), or when you observe separation or a zero cell in the exposure-by-outcome (or covariate-by-outcome) table — the canonical RWE trigger is a rare adverse event with a cell of zero events in one treatment arm or a fully-adjusted model that will not converge. Pre-specify it in the SAP as the primary estimator for rare safety outcomes, or as the fallback estimator invoked by a written rule ("if any exposure-stratum has <5 events or the model fails to converge, switch to Firth penalized likelihood with profile intervals"). It applies to logistic models, conditional logistic models, and Cox proportional-hazards models (penalized partial likelihood), so it covers both binary-outcome and time-to-event RWE.

When NOT to use — and when it is actively misleading or dangerous

. - Do not report Firth Wald confidence intervals or Wald p-values. Under the very sparsity that motivates Firth, the Wald SE-based interval is badly miscalibrated and can hide or exaggerate effects; always use the penalized profile-likelihood interval and penalized LR test. Reporting a Firth point estimate with a default software Wald CI is the most common dangerous error. - Do not use plain Firth when you need calibrated absolute risks / predicted probabilities (e.g., a risk model or standardized/marginal risk difference): the intercept is pulled toward 0.5, inflating predicted probabilities for rare outcomes. Use FLIC/FLAC instead. - It does not fix confounding, selection bias, immortal time, or misclassification. A separation problem caused by a structurally impossible combination (e.g., an outcome that defines the exposure) is a design/data error; Firth will dutifully estimate a finite—but meaningless—coefficient. Diagnose why the cell is empty before penalizing it away. - It is not a substitute for adequate person-time. If the rare event is rare because follow-up is too short or the cohort too small to answer the question, Firth makes the model run but does not manufacture information; the interval will (correctly) be wide. - For very large samples far from separation, there is no advantage and a small null-ward shrinkage cost; use ordinary MLE.

Data-source operational depth

. - Claims (FFS vs Medicare Advantage): Rare safety events with low counts per arm are the dominant Firth use case here. The danger is spurious separation created by data structure rather than biology: if the cohort includes Medicare Advantage person-time, encounter (medical) claims are incomplete because MA plans are capitated, so an outcome that depends on FFS diagnosis/procedure claims can show zero events in a subgroup simply because the claims were never adjudicated to the database — Firth will then "stabilize" a separation that is pure missingness. Restrict to Parts A/B/D FFS enrollees (or the commercial pharmacy+medical benefit) before concluding a zero cell is real. Watch differential competing risks: in elderly claims cohorts, death competes with the event of interest and can differ by exposure, so a near-empty cell in a Cox model may reflect competing mortality, not a protective effect — model the competing risk explicitly and apply Firth to the cause-specific or Fine–Gray model rather than reading the penalized HR as a risk difference. Confirm the rare outcome with continuous enrollment and a clean washout so "no event" is truly observed absence, not unobserved follow-up. - EHR: Encounter-driven capture means a "zero events" subgroup can be patients who left the health system, not patients who did not have the event; loss to follow-up is informative and can manufacture quasi-separation. Use linked claims or a death index to confirm absence of the outcome before penalizing. Site/center effects with few events per site frequently trigger separation in conditional/stratified models — Firth is well suited there, but report the profile interval. - Registry: Adjudicated outcomes are higher quality, but rare-disease registries are small by construction, so low EPV and separation are routine; Firth is often the appropriate primary estimator. Beware registries with eligibility tied to the outcome (e.g., a treatment registry that enrolls at the time of the event), which can create structural separation. - Linked claims–EHR–registry: The richest substrate, but linkage selection and date-reconciliation (order vs fill vs service date) can drop events differentially and produce artificial zero cells; reconcile dates and confirm the linkable subset is not differentially missing the outcome before treating a sparse cell as biological.

Worked claims example

Question: 1-year risk of a rare hepatotoxicity hospitalization with new-use of drug A vs active comparator drug B for the same indication, in a commercial + Medicare FFS database. Cohort: new users (no fill of A or B in a 365-day continuous-enrollment washout), index_date = first qualifying `fill_date` (NDC), arm assigned from that fill. Follow-up from index to the first inpatient claim with the hepatotoxicity ICD-10 code in the primary position, censoring at disenrollment, death, end of data, or 365 days. Suppose the 2×2 is: drug A 6 events / 4,210 initiators; drug B 0 events / 1,905 initiators. Ordinary logistic regression adjusting for age, sex, baseline liver disease (Charlson/Elixhauser proxies from the 365-day lookback), and a high-dimensional PS decile is separated — the comparator arm has zero events, so the MLE odds ratio diverges (software reports an OR in the millions with an infinite SE). Switch to the pre-specified Firth estimator: it returns a finite adjusted log-OR with a penalized profile-likelihood 95% CI (e.g., OR ≈ 5.9, 95% PL CI 0.7–260) and a penalized LR p-value, honestly reflecting that with zero comparator events the data are compatible with a wide range of effects. Before reporting, verify the zero cell is real: confirm no drug-B initiator with MA-only person-time was misclassified as event-free, check that competing death did not remove drug-B patients before they could have the event, and run a sensitivity analysis using the Greenland–Mansournia data-augmentation prior to show the conclusion is not an artifact of the penalty choice.

Interpreting the output

Consider the hepatotoxicity signal above: drug A with 6 liver-injury events in 4,210 initiators versus drug B with 0 events in 1,905 initiators. The Firth-adjusted model returns OR ≈ 5.9 with a 95% profile-likelihood CI of approximately 0.7–260 and a penalized LR p-value.

(1) Formal statistical interpretation. The penalized OR of ≈ 5.9 is the Firth maximum penalized likelihood estimate; it is systematically smaller than the MLE would be if computable (the Jeffreys prior pulls extreme estimates toward one). The profile-likelihood CI — not the Wald interval — must be reported because Wald intervals rely on the normal approximation of the log-OR, which fails completely under separation. The CI of ≈ 0.7–260 is wide because the data contain essentially no comparator-arm information: values of the true OR anywhere in this range are compatible with the observed data.

(2) Practical interpretation for a decision-maker. The Firth OR of ≈ 5.9 is a finite, honest signal that drug A may carry elevated liver-injury risk relative to drug B, but the interval spanning 0.7 to 260 means the data alone cannot rule out a null association or a very large one. This result justifies a safety follow-up in a larger database — it does not support a label change or a formulary restriction on its own. Do not interpret the OR without its full CI; the point estimate is not stable enough to act on in isolation.

Worked example

Scenario

A pharmacoepidemiology team is studying a rare liver injury (hepatotoxicity) in patients who start either Drug A or Drug B. They pull one year of follow-up from a claims database and build a simple 2x2 table to check whether ordinary logistic regression will work before running the full adjusted model. Drug A has 6 patients who experienced the liver injury out of 4,210 starters. Drug B has 0 patients who experienced the liver injury out of 1,905 starters. This is complete separation: the Drug B column has a zero, which means every case came from Drug A. Ordinary logistic regression has no finite answer for this table.

Dataset

2x2 event table from the raw claims cohort: rows are treatment arms, columns are outcome status.

Steps

• Step 1 — Check for separation: scan the table for any zero cell in the outcome columns. Drug B has 0 events. This is complete separation; the ordinary maximum-likelihood odds ratio is undefined (software will return OR in the millions or refuse to converge).

• Step 2 — Confirm the zero is real: verify Drug B starters were enrolled continuously, had a clean washout with no prior liver-injury codes, and that all person-time is from fee-for-service claims where events would be captured. If the zero is an artifact of missing data, fix the data first.

• Step 3 — Apply Firth penalization: instead of maximizing the ordinary log-likelihood, the software maximizes a penalized version that adds a small Jeffreys-prior term. This shifts the score equation just enough to produce a finite estimate.

• Step 4 — Read the result: Firth returns an adjusted OR of approximately 5.9. This means Drug A starters had roughly 6 times the odds of liver injury compared to Drug B starters, after adjustment.

• Step 5 — Report only the profile-likelihood confidence interval (e.g., 95% PL CI: 0.7 to 260), not a Wald interval. The enormous width honestly reflects that with zero Drug B events the data cannot rule out a small or a very large true effect.

Result

Firth-adjusted OR = 5.9 (95% profile-likelihood CI 0.7 to 260). The ordinary maximum-likelihood OR is undefined (infinite) because Drug B has zero events. Firth stabilizes the estimate to a finite value; the wide confidence interval correctly reflects the limits of 6 total events across 6,115 patients.

Runnable example

import pandas as pd
from firthlogist import FirthLogisticRegression

# analytic : one row per cohort member with `arm`, `event`, and pre-index confounders (no toy data created here)
confounders = ["arm", "age", "female", "baseline_liver", "ps_decile"]
X = analytic[confounders].to_numpy()
y = analytic["event"].to_numpy()

# Detect the separation that makes ordinary logistic regression diverge: any exposure arm with zero events.
cell_counts = analytic.groupby("arm")["event"].agg(["sum", "size"])
print("events / n by arm:\n", cell_counts)  # a 0 in the 'sum' column => complete separation

fl = FirthLogisticRegression(
    skip_lrt=False,        # request penalized likelihood-ratio tests, not Wald
    test_vars=0,           # index of `arm` -> profile-likelihood inference for the exposure effect
)
fl.fit(X, y)

# Penalized profile-likelihood CI for the adjusted log-odds-ratio of `arm` (the exposure of interest).
import numpy as np
or_arm = np.exp(fl.coef_[0])
ci_low, ci_high = np.exp(fl.ci_[0])     # ci_ holds penalized profile-likelihood limits
print(f"Adjusted OR (drug vs comparator) = {or_arm:.2f} "
      f"(95% profile-likelihood CI {ci_low:.2f} to {ci_high:.2f}); "
      f"penalized LR p = {fl.pvals_[0]:.4f}")

library(logistf)

## Diagnose separation: a zero in any exposure-by-outcome cell breaks ordinary glm().
print(table(analytic$arm, analytic$event))

## Firth penalized logistic regression; profile-likelihood CIs + penalized LR tests are the default.
fit <- logistf(event ~ arm + age + female + baseline_liver + factor(ps_decile),
               data = analytic)
summary(fit)                       # coef, profile-likelihood 95% CI, penalized LR p-values
exp(cbind(OR = coef(fit),          # adjusted odds ratios with penalized profile-likelihood limits
          `2.5%`  = fit$ci.lower,
          `97.5%` = fit$ci.upper))

## Time-to-event variant: Firth penalty on the Cox partial likelihood (monotone-likelihood fix).
library(coxphf)
cox_fit <- coxphf(Surv(fu_days, event) ~ arm + age + female + baseline_liver,
                  data = analytic)
summary(cox_fit)                   # penalized HR with profile-likelihood CI for time-to-event outcomes