Empirical Calibration with Negative Controls

In plain language

Empirical calibration checks whether a study's analysis pipeline is already producing systematically wrong answers even for drug-outcome pairs where the true answer is known to be null, and then uses that pattern of wrongness to adjust the final result. You pick many outcome measures that the drug in question could not plausibly cause -- for example, ingrown toenail or cataract -- run the identical analysis on each of them, and collect all those estimates. If they cluster away from the true null, the pipeline itself carries a hidden bias; the method measures that bias and recalculates your p-value and confidence interval accordingly, making the result more honest. A key caveat is that this works only when you have at least 20 or more of those known-null pairs and they all share the same sources of error as your real question.

Empirical calibration

treats the effect estimates from a large panel of negative controls — exposure-outcome pairs for which the true relative effect is known (or assumed) to be null — as draws from the systematic-error distribution that the study's design, data source, and analytic pipeline impose on every estimate, including the target. Instead of using one or two negative controls as pass/fail falsification tests, it pools tens to hundreds of them, fits an empirical null (typically a Gaussian on the log-rate-ratio scale with mean `mu` and SD `sigma`, estimated by maximum likelihood while propagating each control's own standard error), and then evaluates the target estimate against that empirical null rather than against the theoretical null of mean 0, variance 0. The recalibrated p-value is the tail probability of the target's log estimate under the fitted systematic-error distribution; the recalibrated confidence interval additionally uses synthetic positive controls (negative controls into which a known relative risk has been injected) to learn how bias and its variance scale across the true effect size, then widens and shifts the interval accordingly (Schuemie 2014; Schuemie 2018).

Core conceptual distinction

A conventional p-value asks "how surprising is this estimate if the true effect is null and the only error is random sampling?" Empirical calibration replaces the second clause: it asks "how surprising is this estimate relative to the estimates the same pipeline produces for things we know are null?" The quantity being modeled is not the target effect but the systematic error `theta - log(RR_true)` of the procedure. Two estimands must be kept distinct and pre-specified. (1) The calibrated p-value tests the null using the empirical null `N(mu, sigma^2)`; it corrects for the mean and spread of residual confounding/measurement bias but assumes that bias does not vary with the true effect size. (2) The calibrated confidence interval uses a systematic error model fit to both negative and positive controls, allowing `mu` and `sigma` to depend on the true log-RR, and yields an interval with the nominal empirical coverage (e.g., 95% of calibrated CIs cover the true effect across the control panel). Calibration recalibrates inference; it does not change the point estimate and is not a substitute for a sound design — it diagnoses and partially corrects the residual error a good design leaves behind, and it inflates uncertainty rather than manufacturing precision.

Pros, cons, and trade-offs

- vs single/few negative controls (falsification tests): the standard practice of running one or two negative controls and eyeballing whether they are "null enough" has no calibrated decision rule and no way to translate an observed control bias into corrected inference for the target. Empirical calibration turns the same controls into a quantitative, pre-specifiable correction with an interpretable empirical null. Cost: it needs many credible controls (rule of thumb ≥20-50; CI calibration needs positive controls too), and it assumes the controls share the target's systematic-error structure. Prefer empirical calibration in high-throughput claims/EHR or network studies where a control library can be curated; prefer a handful of falsification controls when only a few defensible controls exist. - vs E-value / array-based sensitivity analysis: the E-value asks how strong an unmeasured confounder would have to be to explain away the estimate; it is a hypothetical stress test requiring no control data. Empirical calibration instead measures the net residual bias the pipeline actually exhibits on nulls. Cost: the E-value is model-free and needs no controls but says nothing about measurement error, misclassification, or selection that calibration captures jointly; calibration captures the aggregate of all such biases but cannot attribute them to a specific source. - vs probabilistic bias analysis (PBA) / bias-parameter QBA: PBA places priors on specific bias parameters (sensitivity/specificity, confounder prevalence, RR) and propagates them. Empirical calibration is data-driven and parameter-free: it learns the net bias from observed nulls instead of eliciting bias priors. Cost: PBA can model biases for which no negative controls exist and can decompose by mechanism; calibration cannot model a bias the controls do not share and gives no mechanistic decomposition. They are complements, not substitutes — calibration for net residual bias you can observe, PBA for named biases you must assume.

When to use

High-throughput observational effect-estimation where the same design/analysis is applied uniformly across many outcomes or exposures (OHDSI-style network studies, signal screening, large claims/EHR safety studies); whenever a credible library of ≥20-50 negative controls can be assembled with reliable phenotype algorithms and adequate counts; when reviewers/regulators want residual systematic error quantified rather than merely asserted; and when the design is already defensible (active comparator, new-user, washout) so that calibration is correcting a small residual rather than rescuing a broken comparison. Use CI calibration (with positive controls) whenever you report intervals, not just significance.

When NOT to use — and when it is actively misleading or dangerous

- Too few or contaminated controls. With <20 controls the empirical null is estimated with large uncertainty and the correction is unstable; if some "negative" controls are in fact non-null (the exposure really does affect them), the null is biased and calibration under-corrects or shifts the wrong way. Curating the control panel is the load-bearing assumption — a panel chosen for convenience can be worse than no calibration. - Controls do not share the target's systematic-error structure. If negative controls are drawn from different outcome-detection pathways (e.g., controls are inpatient-coded events but the target is an outpatient-coded event with different surveillance), the empirical null estimates the wrong bias distribution. Calibration silently licenses a contaminated comparison — this is the dangerous failure: a tidy "calibrated" CI that imports the bias of irrelevant controls. - As a substitute for design. Calibration cannot fix confounding by indication, immortal time, or a wrong comparator; if the design is broken the empirical null will be wide and off-center, and a calibrated CI that still excludes the null is more likely to reflect a non-shared bias than a real effect. Calibration is a final-mile diagnostic, not a rescue. - Differential bias by true effect size without positive controls. Using p-value calibration alone (assuming bias is constant across effect sizes) when bias actually scales with the effect will produce miscalibrated intervals; CI calibration with synthetic positive controls is required, and synthetic positives can themselves be unrealistic if the injection mechanism does not resemble a true causal effect.

Data-source operational depth

- Administrative claims (FFS): the natural home — large, uniform, and amenable to high-throughput control panels built from condition concept sets. Failure modes: (i) Medicare Advantage / capitated person-time lacks complete FFS claims, so control-outcome counts are differentially undercounted for MA enrollees; restrict the control analyses to the same FFS-observable person-time used for the target, or the empirical null will reflect missingness rather than bias. (ii) Differential competing risks by exposure in elderly claims (a comparator with higher background mortality removes person-time before the control outcome can be coded) shifts negative-control rate ratios away from null in a direction the target shares — this is good (calibration captures it) only if the controls experience the same competing risk; pick controls with similar event timing. (iii) Low-count controls produce unstable per-control standard errors; require a minimum cell count and use the MLE that weights by precision rather than treating each control equally. - EHR: site-specific coding, referral patterns, and visit-driven capture make systematic error heterogeneous across sites. A single pooled empirical null can be dominated by one site's idiosyncratic miscoding. Calibrate within database/site and meta-analyze the calibrated estimates, or include site as a stratifier in the control analyses. Note/lab-derived phenotypes change negative-control specificity, so a control that is "null" in claims may not be in EHR. - Registry: strong outcome adjudication makes individual controls cleaner but the number of curatable controls is usually small, undermining the empirical-null fit; registries are better as the outcome layer in a linked design than as a standalone calibration substrate. - Linked claims-EHR-registry: best of both — EHR/registry for clean control phenotypes and claims for complete person-time — but linkage selection means the calibration panel must be drawn from the same linkable subset as the target, and order/fill/service date discrepancies must be reconciled before counting control events.

Worked claims example

Question: is initiation of drug A vs active comparator B associated with acute myocardial infarction among adults with the shared indication, in a commercial + Medicare FFS database, using a new-user active-comparator design with high-dimensional PS adjustment. (1) Target analysis: build the cohort with 365-day continuous FFS-observable enrollment and washout, time zero at first qualifying fill, and estimate the AMI hazard ratio with a Cox model on the PS-matched set — say `HR = 1.30`, 95% CI 1.05-1.61, p = 0.017. (2) Negative-control panel: select ~50 outcomes with no plausible causal link to either drug (e.g., ingrowing nail, contusion, cataract — each a validated concept set with adequate counts) and run the identical pipeline (same washout, same FFS-observable person-time excluding MA-only spans, same PS model, same Cox specification) for each, yielding 50 pairs of `(log_hr, se)`. (3) Fit the empirical null: by MLE, the controls center at `mu = 0.08` (log scale) with extra dispersion `sigma = 0.10` beyond sampling error — evidence of mild residual confounding biasing estimates ~8% upward. (4) Calibrate the p-value: evaluate the target's `log(1.30)=0.262` against `N(0.08, 0.10^2 + se_target^2)`; the calibrated p rises from 0.017 to ~0.08 — no longer "significant" once the pipeline's own null behavior is accounted for. (5) Synthetic positive controls + CI calibration: inject known HRs (1.5, 2, 4) into negative controls by adding the corresponding extra outcomes, fit the systematic-error model across true effect sizes, and recompute the interval: calibrated 95% CI ≈ 0.92-1.84. (6) Report both the crude and calibrated inference and the empirical-null parameters as a diagnostic; a `mu` far from 0 or large `sigma` is a flag that the design, not the calibration, needs revisiting.

Interpreting the output

From the worked example: naive HR = 1.30 (95% CI 1.09–1.56, p = 0.017). Eight negative controls yield empirical null N(mu = 0.08, sigma = 0.10). Calibrated p ≈ 0.08; calibrated 95% CI ≈ 0.92–1.84.

(1) Formal interpretation. The calibrated p-value is the probability of observing a log-HR as extreme as log(1.30) = 0.262 under the empirical null distribution N(0.08, 0.10² + se_target²), which models the study pipeline's observed behavior when the true effect is zero. It is not a frequentist p-value over hypothetical replications of a fixed data-generating process — it is inference benchmarked against this design's own null-control behavior. The calibrated CI is similarly re-centered (shifted away from 0) and widened to absorb the pipeline's systematic error; it is not a conventional Wald interval. The empirical null parameters (mu = 0.08, sigma = 0.10) are themselves estimated with uncertainty from only eight controls, so both calibrated quantities carry imprecision from that estimation step.

(2) Practical interpretation. The naive analysis declared statistical significance (p = 0.017), but after accounting for the pipeline's tendency to over-estimate by ≈ 8% in log-HR units and its extra dispersion beyond sampling variability, the calibrated p rises to ≈ 0.08 and the CI widens to span nearly the null. A regulator or payer should treat the calibrated result — not the naive one — as the study's actual inferential claim. A large mu or sigma in the empirical null is itself diagnostic: it signals that the design or covariate adjustment needs reconsidering before calibration can rescue it.

Worked example

Scenario

A team runs a new-user active-comparator study asking whether Drug A raises the risk of heart attack (acute myocardial infarction, AMI) compared with Drug B in a claims database. Their main result is a hazard ratio of 1.30 with a naive p-value of 0.017 -- apparently statistically significant. Before trusting that finding, they run the identical analysis pipeline on eight negative-control outcomes: conditions like ingrown toenail, contusion, and cataract that Drug A cannot plausibly cause. If the pipeline were unbiased, every negative-control hazard ratio should hover near 1.0 (log scale: near 0). Instead, the team finds that all eight cluster above 1.0, suggesting the pipeline systematically inflates estimates by roughly 8 percent on the log scale. Calibrating the AMI result against this observed bias pattern changes the conclusion.

Dataset

Negative-control estimates from running the identical pipeline on eight known-null outcomes. Log HR near 0 would be expected if there were no bias; values above 0 reveal systematic upward drift.

Steps

• Step 1 -- Spot the pattern: every single negative-control log HR is positive (range 0.06 to 0.10) even though the true effect for each should be 0. This is not random noise; all eight point in the same direction.

• Step 2 -- Fit the empirical null: maximum-likelihood estimation across all eight pairs gives a null distribution centered at mu = 0.08 (log scale) with a spread of sigma = 0.10. This means the pipeline tends to inflate log hazard ratios by about 0.08 on average, with modest extra variability.

• Step 3 -- Interpret the empirical null: a naive analysis assumes the pipeline's null is centered at 0 with no extra spread. The empirical null says the real center is 0.08, not 0 -- so the bar the AMI result must clear is higher.

• Step 4 -- Calibrate the AMI p-value: the AMI log HR is log(1.30) = 0.262. Instead of asking how far 0.262 is from 0 (the naive test), calibration asks how far 0.262 is from 0.08 in units of the combined uncertainty (sampling error plus the sigma = 0.10 from the empirical null). The adjusted distance is smaller, and the calibrated p-value rises from 0.017 to approximately 0.08 -- no longer below the conventional 0.05 threshold.

• Step 5 -- Calibrate the confidence interval: the interval is re-centered by subtracting the empirical bias (0.08) and widened to account for the extra spread (sigma = 0.10). The naive 95% CI of 1.05 to 1.61 becomes approximately 0.92 to 1.84 after calibration -- now spanning 1.0, consistent with no effect.

Result

Naive result: HR = 1.30, 95% CI 1.05-1.61, p = 0.017 (appears significant). Empirical null from 8 negative controls: mu = 0.08, sigma = 0.10 (pipeline inflates all estimates ~8% upward on the log scale). Calibrated result: p = 0.08 (no longer significant), calibrated 95% CI approximately 0.92-1.84 (now includes 1.0). The apparent signal for AMI is explained by the systematic upward drift the pipeline imposes on every estimate, not by a true drug effect.

Runnable example

import numpy as np
from scipy.optimize import minimize
from scipy.stats import norm

def fit_empirical_null(log_rr: np.ndarray, se: np.ndarray) -> tuple[float, float]:
    # MLE of mu, sigma for theta_i ~ N(mu, sigma^2) observed with sampling noise:
    # log_rr_i ~ N(mu, sigma^2 + se_i^2).  (Schuemie 2014, eq. for the empirical null.)
    def neg_ll(p):
        mu, log_sigma = p
        var = np.exp(2 * log_sigma) + se ** 2
        return 0.5 * np.sum(np.log(2 * np.pi * var) + (log_rr - mu) ** 2 / var)
    start = [float(np.mean(log_rr)), np.log(np.std(log_rr, ddof=1) + 1e-6)]
    res = minimize(neg_ll, start, method="Nelder-Mead")
    mu, log_sigma = res.x
    return float(mu), float(np.exp(log_sigma))

def calibrate_p(target_log_rr: float, target_se: float,
                mu: float, sigma: float) -> float:
    # Two-sided tail probability of the target under the empirical null,
    # adding the target's own sampling variance to the null variance.
    sd = np.sqrt(sigma ** 2 + target_se ** 2)
    z = (target_log_rr - mu) / sd
    return float(2 * norm.sf(abs(z)))

def calibrate_ci(target_log_rr: float, target_se: float,
                 mu: float, sigma: float, level: float = 0.95) -> tuple[float, float]:
    # Gaussian-approx calibrated interval: re-center by the null mean and widen by
    # the systematic-error SD. (Use the OHDSI systematic-error model for the full,
    # effect-size-dependent version with positive controls.)
    sd = np.sqrt(sigma ** 2 + target_se ** 2)
    crit = norm.ppf(1 - (1 - level) / 2)
    lo, hi = (target_log_rr - mu) - crit * sd, (target_log_rr - mu) + crit * sd
    return float(np.exp(lo)), float(np.exp(hi))

# Usage on a curated control panel + a target estimate:
# mu, sigma = fit_empirical_null(controls["log_rr"].to_numpy(), controls["se"].to_numpy())
# cal_p = calibrate_p(target_log_rr, target_se, mu, sigma)
# cal_lo, cal_hi = calibrate_ci(target_log_rr, target_se, mu, sigma)

library(EmpiricalCalibration)

# --- p-value calibration: empirical null from negative controls only ---
null <- fitNull(negatives$logRr, negatives$seLogRr)          # MLE of (mean, sd) of the null
cal_p <- calibrateP(null, target_logRr, target_seLogRr)      # two-sided calibrated p-value

# --- confidence-interval calibration: systematic-error model (negatives + positives) ---
model <- fitSystematicErrorModel(
  logRr    = c(negatives$logRr,    positives$logRr),
  seLogRr  = c(negatives$seLogRr,  positives$seLogRr),
  trueLogRr = c(rep(0, nrow(negatives)), positives$trueLogRr)
)
cal_ci <- calibrateConfidenceInterval(target_logRr, target_seLogRr, model)
# cal_ci$logRr, cal_ci$logLb95Rr, cal_ci$logUb95Rr  -> exponentiate for RR scale

# --- diagnostic: empirical-null plot and leave-one-out coverage on the controls ---
plotCalibrationEffect(negatives$logRr, negatives$seLogRr)
eval <- evaluateCalibration95ci(negatives$logRr, negatives$seLogRr)  # should be ~0.95