Firth Penalized Regression

Firth penalized regression

modifies the likelihood of a logistic or Cox model by adding a penalty term equal to one-half the log-determinant of the Fisher information (the Jeffreys invariant prior). Maximizing this penalized log-likelihood shifts the score equations by a term that exactly cancels the leading O(1/n) bias of the ordinary maximum likelihood estimate (MLE).

The practical consequence in real-world evidence (RWE) is twofold: (1) coefficients are bias-reduced in small or imbalanced samples, and (2) — crucially for pharmacoepidemiology — estimates remain finite and unique even under complete or quasi-complete separation, the situation where an exposure or covariate perfectly (or nearly perfectly) predicts the outcome and the ordinary MLE diverges to ±infinity with an infinite standard error.

Firth was conceived as a bias-reduction device (Firth 1993); Heinze and Schemper (2002) recognized it as the principled fix for separation, which is the reason it appears constantly in rare-outcome comparative-safety analyses.

Core conceptual distinction

. Firth is a finite-sample / sparse-data correction, not an alternative estimand. It targets the same conditional log–odds ratio or log–hazard ratio as ordinary logistic/Cox regression; it does not change what is being estimated, only how it is estimated when the data are too thin for the asymptotic MLE to behave. Distinguish it from three neighbours that are often conflated with it.

1. Exact logistic regression conditions on sufficient statistics and gives exact small-sample inference, but is computationally intractable for the multi-covariate, high-dimensional confounder sets typical of claims studies — Firth is the scalable approximation that keeps continuous covariates and many terms.
2. Ridge/LASSO penalize the size of coefficients toward zero to manage variance/selection; Firth penalizes via the information matrix to remove bias and guarantee finiteness, and (with the appropriate profile-penalized-likelihood interval) it is invariant to predictor scaling in a way ridge is not.
3. Exact/Poisson or rare-event corrections (e.g., King–Zeng) address a related rare-events problem but with a different correction; Firth is the more general, model-internal solution.

A subtle but important caveat: standard Firth biases the intercept (and therefore predicted probabilities) toward 1/2, so for absolute-risk prediction in rare outcomes use FLIC (Firth logistic regression with intercept correction) or FLAC; for the comparative log–odds ratio that drives most RWE, plain Firth is appropriate.

Pros, cons, and trade-offs

(specific and comparative).

• vs ordinary (unpenalized) logistic/Cox MLE: Firth always returns a finite estimate and a usable confidence interval under separation, where the MLE returns ±infinity, an absurd odds ratio (e.g., OR = 4.7e8), or a model that silently fails to converge. Even without separation it has smaller mean-squared error at low events-per-variable (EPV). Cost: it is a shrinkage estimator, so in large, well-separated-from-the-boundary samples it adds a touch of bias toward the null relative to the (then-unbiased) MLE, and Wald intervals are unreliable — you must use penalized profile-likelihood intervals and penalized likelihood-ratio tests, not Wald z/SE.
• vs exact logistic regression: Firth scales to realistic claims confounder sets and continuous covariates and is far faster; exact methods give guaranteed small-sample coverage but become infeasible beyond a handful of categorical predictors.
• vs dropping/collapsing the offending variable or using a non-informative prior in a Bayesian model: Firth keeps the variable in the model and is reproducible and deterministic; the Bayesian data-augmentation prior of Greenland and Mansournia (a weakly informative prior on the log-OR) is an excellent, often preferable alternative when you want a transparently chosen prior or posterior interval, and is conceptually close to Firth.
• vs ridge/LASSO penalization: Firth does not select variables and does not require cross-validating a tuning parameter; choose ridge/LASSO when the goal is high-dimensional prediction or selection, Firth when the goal is a (near-)unbiased, finite effect estimate under sparsity.

When NOT to use — and when it is actively misleading or dangerous

.

• Do not report Firth Wald confidence intervals or Wald p-values. Under the very sparsity that motivates Firth, the Wald SE-based interval is badly miscalibrated and can hide or exaggerate effects; always use the penalized profile-likelihood interval and penalized LR test. Reporting a Firth point estimate with a default software Wald CI is the most common dangerous error.
• Do not use plain Firth when you need calibrated absolute risks / predicted probabilities (e.g., a risk model or standardized/marginal risk difference): the intercept is pulled toward 0.5, inflating predicted probabilities for rare outcomes. Use FLIC/FLAC instead.
• It does not fix confounding, selection bias, immortal time, or misclassification. A separation problem caused by a structurally impossible combination (e.g., an outcome that defines the exposure) is a design/data error; Firth will dutifully estimate a finite—but meaningless—coefficient. Diagnose why the cell is empty before penalizing it away.
• It is not a substitute for adequate person-time. If the rare event is rare because follow-up is too short or the cohort too small to answer the question, Firth makes the model run but does not manufacture information; the interval will (correctly) be wide.
• For very large samples far from separation, there is no advantage and a small null-ward shrinkage cost; use ordinary MLE.

Data-source operational depth

.

• Claims (FFS vs Medicare Advantage): Rare safety events with low counts per arm are the dominant Firth use case here. The danger is spurious separation created by data structure rather than biology: if the cohort includes Medicare Advantage person-time, encounter (medical) claims are incomplete because MA plans are capitated, so an outcome that depends on FFS diagnosis/procedure claims can show zero events in a subgroup simply because the claims were never adjudicated to the database — Firth will then "stabilize" a separation that is pure missingness. Restrict to Parts A/B/D FFS enrollees (or the commercial pharmacy+medical benefit) before concluding a zero cell is real. Watch differential competing risks: in elderly claims cohorts, death competes with the event of interest and can differ by exposure, so a near-empty cell in a Cox model may reflect competing mortality, not a protective effect — model the competing risk explicitly and apply Firth to the cause-specific or Fine–Gray model rather than reading the penalized HR as a risk difference. Confirm the rare outcome with continuous enrollment and a clean washout so "no event" is truly observed absence, not unobserved follow-up.
• EHR: Encounter-driven capture means a "zero events" subgroup can be patients who left the health system, not patients who did not have the event; loss to follow-up is informative and can manufacture quasi-separation. Use linked claims or a death index to confirm absence of the outcome before penalizing. Site/center effects with few events per site frequently trigger separation in conditional/stratified models — Firth is well suited there, but report the profile interval.
• Registry: Adjudicated outcomes are higher quality, but rare-disease registries are small by construction, so low EPV and separation are routine; Firth is often the appropriate primary estimator. Beware registries with eligibility tied to the outcome (e.g., a treatment registry that enrolls at the time of the event), which can create structural separation.
• Linked claims–EHR–registry: The richest substrate, but linkage selection and date-reconciliation (order vs fill vs service date) can drop events differentially and produce artificial zero cells; reconcile dates and confirm the linkable subset is not differentially missing the outcome before treating a sparse cell as biological.

Worked claims example

Question: 1-year risk of a rare hepatotoxicity hospitalization with new-use of drug A vs active comparator drug B for the same indication, in a commercial + Medicare FFS database. Cohort: new users (no fill of A or B in a 365-day continuous-enrollment washout), index_date = first qualifying `fill_date` (NDC), arm assigned from that fill. Follow-up from index to the first inpatient claim with the hepatotoxicity ICD-10 code in the primary position, censoring at disenrollment, death, end of data, or 365 days.

Suppose the 2×2 is: drug A 6 events / 4,210 initiators; drug B 0 events / 1,905 initiators. Ordinary logistic regression adjusting for age, sex, baseline liver disease (Charlson/Elixhauser proxies from the 365-day lookback), and a high-dimensional PS decile is separated — the comparator arm has zero events, so the MLE odds ratio diverges (software reports an OR in the millions with an infinite SE).

Switch to the pre-specified Firth estimator: it returns a finite adjusted log-OR with a penalized profile-likelihood 95% CI (e.g., OR ≈ 5.9, 95% PL CI 0.7–260) and a penalized LR p-value, honestly reflecting that with zero comparator events the data are compatible with a wide range of effects.

Before reporting, verify the zero cell is real: confirm no drug-B initiator with MA-only person-time was misclassified as event-free, check that competing death did not remove drug-B patients before they could have the event, and run a sensitivity analysis using the Greenland–Mansournia data-augmentation prior to show the conclusion is not an artifact of the penalty choice.

Interpreting the output

Consider the hepatotoxicity signal above: drug A with 6 liver-injury events in 4,210 initiators versus drug B with 0 events in 1,905 initiators. The Firth-adjusted model returns OR ≈ 5.9 with a 95% profile-likelihood CI of approximately 0.7–260 and a penalized LR p-value.

(1) Formal statistical interpretation. The penalized OR of ≈ 5.9 is the Firth maximum penalized likelihood estimate; it is systematically smaller than the MLE would be if computable (the Jeffreys prior pulls extreme estimates toward one). The profile-likelihood CI — not the Wald interval — must be reported because Wald intervals rely on the normal approximation of the log-OR, which fails completely under separation. The CI of ≈ 0.7–260 is wide because the data contain essentially no comparator-arm information: values of the true OR anywhere in this range are compatible with the observed data.

(2) Practical interpretation for a decision-maker. The Firth OR of ≈ 5.9 is a finite, honest signal that drug A may carry elevated liver-injury risk relative to drug B, but the interval spanning 0.7 to 260 means the data alone cannot rule out a null association or a very large one. This result justifies a safety follow-up in a larger database — it does not support a label change or a formulary restriction on its own. Do not interpret the OR without its full CI; the point estimate is not stable enough to act on in isolation.

Decision diagram

flowchart TD
  Fit[Fit fully-adjusted logistic / Cox model on the RWE cohort] --> Q{Separation, zero cell,<br/>or EPV < ~10?}
  Q -- No --> MLE[Use ordinary maximum likelihood<br/>report Wald or LR inference]
  Q -- Yes --> Real{Is the empty cell real?<br/>FFS-only person-time, true washout,<br/>not competing death or MA missingness}
  Real -- No --> Fix[Fix the data/design problem first<br/>restrict to FFS, model competing risk, reconcile linkage]
  Real -- Yes --> Firth[Fit Firth penalized likelihood<br/>logistic / Cox / cause-specific]
  Firth --> Inf[Report penalized PROFILE-LIKELIHOOD CIs<br/>and penalized LR tests -- never Wald]
  Inf --> Est{Estimand = conditional OR/HR<br/>or absolute / standardized risk?}
  Est -- OR / HR --> Plain[Plain Firth is appropriate]
  Est -- Absolute risk --> FLIC[Use FLIC / FLAC<br/>intercept-corrected Firth for calibration]

flowchart LR
  MLE["Ordinary MLE log-likelihood<br/>diverges under separation<br/>(estimate to +/- infinity)"] -->|"add Jeffreys penalty<br/>+ 0.5 * log det I(beta)"| Pen["Penalized log-likelihood"]
  Pen --> Score["Modified score equation<br/>cancels O(1/n) bias"]
  Score --> Out["Finite, unique, bias-reduced<br/>OR / HR estimate"]
  Out --> CI["Inference by penalized<br/>profile likelihood (not Wald)"]

Worked example

Scenario

A pharmacoepidemiology team is studying a rare liver injury (hepatotoxicity) in patients who start either Drug A or Drug B. They pull one year of follow-up from a claims database and build a simple 2x2 table to check whether ordinary logistic regression will work before running the full adjusted model. Drug A has 6 patients who experienced the liver injury out of 4,210 starters. Drug B has 0 patients who experienced the liver injury out of 1,905 starters. This is complete separation: the Drug B column has a zero, which means every case came from Drug A. Ordinary logistic regression has no finite answer for this table.

Dataset

2x2 event table from the raw claims cohort: rows are treatment arms, columns are outcome status.

Steps

Result

Firth-adjusted OR = 5.9 (95% profile-likelihood CI 0.7 to 260). The ordinary maximum-likelihood OR is undefined (infinite) because Drug B has zero events. Firth stabilizes the estimate to a finite value; the wide confidence interval correctly reflects the limits of 6 total events across 6,115 patients.

Trade-offs

Runnable example

import pandas as pd
from firthlogist import FirthLogisticRegression

# analytic : one row per cohort member with `arm`, `event`, and pre-index confounders (no toy data created here)
confounders = ["arm", "age", "female", "baseline_liver", "ps_decile"]
X = analytic[confounders].to_numpy()
y = analytic["event"].to_numpy()

# Detect the separation that makes ordinary logistic regression diverge: any exposure arm with zero events.
cell_counts = analytic.groupby("arm")["event"].agg(["sum", "size"])
print("events / n by arm:\n", cell_counts)  # a 0 in the 'sum' column => complete separation

fl = FirthLogisticRegression(
    skip_lrt=False,        # request penalized likelihood-ratio tests, not Wald
    test_vars=0,           # index of `arm` -> profile-likelihood inference for the exposure effect
)
fl.fit(X, y)

# Penalized profile-likelihood CI for the adjusted log-odds-ratio of `arm` (the exposure of interest).
import numpy as np
or_arm = np.exp(fl.coef_[0])
ci_low, ci_high = np.exp(fl.ci_[0])     # ci_ holds penalized profile-likelihood limits
print(f"Adjusted OR (drug vs comparator) = {or_arm:.2f} "
      f"(95% profile-likelihood CI {ci_low:.2f} to {ci_high:.2f}); "
      f"penalized LR p = {fl.pvals_[0]:.4f}")

library(logistf)

## Diagnose separation: a zero in any exposure-by-outcome cell breaks ordinary glm().
print(table(analytic$arm, analytic$event))

## Firth penalized logistic regression; profile-likelihood CIs + penalized LR tests are the default.
fit <- logistf(event ~ arm + age + female + baseline_liver + factor(ps_decile),
               data = analytic)
summary(fit)                       # coef, profile-likelihood 95% CI, penalized LR p-values
exp(cbind(OR = coef(fit),          # adjusted odds ratios with penalized profile-likelihood limits
          `2.5%`  = fit$ci.lower,
          `97.5%` = fit$ci.upper))

## Time-to-event variant: Firth penalty on the Cox partial likelihood (monotone-likelihood fix).
library(coxphf)
cox_fit <- coxphf(Surv(fu_days, event) ~ arm + age + female + baseline_liver,
                  data = analytic)
summary(cox_fit)                   # penalized HR with profile-likelihood CI for time-to-event outcomes

/* Surface the separation: a zero events cell in one arm is the trigger for Firth. */
proc freq data=work.analytic;
  tables arm*event / norow nocol nopercent;
run;

/* Firth penalized LOGISTIC regression for a rare binary outcome.                       */
/* CLODDS=PL gives profile-likelihood odds-ratio CIs (NOT the invalid Wald CIs);        */
/* CLPARM=PL gives profile-likelihood parameter CIs. Inference is penalized-likelihood. */
proc logistic data=work.analytic;
  class female(ref='0') ps_decile / param=ref;
  model event(event='1') = arm age female baseline_liver ps_decile
        / firth clodds=pl clparm=pl;
  oddsratio arm;                  /* adjusted OR for drug vs comparator, profile-likelihood CI */
run;

/* Firth penalized COX model for the time-to-event version (monotone-likelihood fix).   */
proc phreg data=work.analytic;
  class female(ref='0') / param=ref;
  model fu_days*event(0) = arm age female baseline_liver / firth risklimits=pl;
  hazardratio arm;                /* penalized HR with profile-likelihood CI */
run;

Citations

1. [1]Firth D. Bias reduction of maximum likelihood estimates. Biometrika. 1993;80(1):27-38.
2. [2]Greenland S, Mansournia MA, Altman DG. Sparse data bias: a problem hiding in plain sight. BMJ. 2016;352:i1981.
3. [3]Concato J, Peduzzi P, Holford TR, Feinstein AR. Importance of events per independent variable in proportional hazards analysis I. Background, goals, and general strategy. Journal of Clinical Epidemiology. 1995;48(12):1495-1501.

1. [4]Heinze G, Schemper M. A solution to the problem of separation in logistic regression. Statistics in Medicine. 2002;21(16):2409-2419.