Negative Control Outcomes

In plain language

A negative control outcome is an event your study drug could not plausibly cause — something like a broken arm or appendicitis — that you run through your exact same analysis as a sanity check. Because the true effect of the drug on that outcome must be zero, any non-zero result you get back is measuring systematic error, not a real drug effect. If your negative control analysis comes back clean (no effect detected), that is evidence your main analysis is also free of the bias you were worried about; if it shows a spurious effect, you know your main result is tainted and must be interpreted with caution.

A negative control outcome (NCO) is an event the study exposure should not cause through any direct or indirect causal path, yet one that is subject to the same sources of residual systematic error that threaten the primary endpoint: unmeasured confounding, healthy-user / healthy-adherer selection, depletion of susceptibles, differential surveillance or coding intensity, and care-seeking behavior. You run the identical analysis used for the primary outcome — same cohort, same time zero, same covariates, same propensity-score model, same weights or matched set, same censoring rules — but swap the dependent variable for the NCO. Because the true exposure-NCO effect is null by construction, any non-null NCO association is a direct, empirical signal that the design and adjustment left detectable bias behind. The canonical real-world demonstration is influenza-vaccine effectiveness in seniors: vaccinated elders appear to have far lower all-cause mortality before influenza season ever starts (Jackson 2006) — an effect the vaccine cannot produce, exposing healthy-vaccinee confounding that contaminates the in-season estimate too.

Core conceptual distinction

An NCO is defined by two simultaneous properties that pull in opposite directions and must both hold. (1) Null causal effect: the exposure has no effect on the NCO — no pharmacology, no behavioral pathway, no detection pathway tied to the drug. (2) Shared bias structure: the NCO is moved by the same unmeasured or imperfectly-controlled factors that move the primary outcome. The diagnostic power comes from the gap between what the model estimates (should be null) and what it actually estimates (the bias). Critically distinguish the NCO from a negative control exposure (NCE) — an exposure that cannot affect the primary outcome but shares its confounding (the symmetric falsification). NCOs are usually easier to find in claims (thousands of diagnosis codes) and are easier to match on surveillance/coding intensity; NCEs are often more directly analogous to the confounding-by-indication structure of the study exposure. Also distinguish the estimand: a single pre-specified NCO is a falsification test (binary credibility signal), whereas a curated panel of dozens of NCOs supports empirical calibration — estimating the systematic-error distribution and recalibrating the primary p-value and confidence interval (Schuemie 2014, 2018). The two uses require different numbers of controls and different assumptions; do not conflate them.

Pros, cons, and trade-offs

- vs the E-value (e-value-sensitivity-analysis): The E-value reports the minimum confounding strength on the risk-ratio scale needed to explain away the result, under an assumption-only model that addresses unmeasured confounding alone. The NCO is empirical — it uses the actual data and the exact analytic pipeline, and it detects bias from sources the E-value ignores (surveillance, coding, care-seeking, selection). Cost: the NCO gives a qualitative pass/fail (or a calibrated interval with a panel), not a single interpretable sensitivity bound; its power depends on NCO frequency. Use both — they answer different questions. - vs probabilistic / quantitative bias analysis (unmeasured-confounding-probabilistic-bias-analysis-rwe): QBA propagates external bias parameters (sensitivity/specificity, confounder prevalence, bias factors) into a corrected estimate or simulation interval for the primary parameter. The NCO does not correct anything — it falsifies. QBA needs credible external parameters; the NCO needs only a credible control outcome. They are complementary: NCO to detect, QBA to quantify. - vs a validation substudy / chart review: A validation substudy can actually correct outcome misclassification using gold-standard adjudication, but it is expensive and slow. A well-powered NCO test is cheap and uses data you already have — at the price of falsifying rather than fixing. - vs empirical calibration with NCEs alone: NCOs can be tuned to share the primary outcome's exact detection intensity (e.g., another inpatient primary diagnosis), which an NCE cannot; but a poorly chosen NCO (traumatic injury, elective surgery) may not share the confounding-by-indication structure of a specific drug-outcome pair. Best practice combines NCOs and NCEs (the "negative control pair").

When to use

- After a defensible primary design (new-user active-comparator or target-trial emulation) when the result is surprising, policy-relevant, or regulatory-facing, and you want a data-driven check that residual bias was removed. - When the primary analysis leans on high-dimensional or ML-based confounding control (hdPS, large-scale PS) and you need empirical evidence — not just balance tables — that adjustment worked. - In multi-database / OHDSI-style network studies, where a curated NCO panel enables empirical calibration so that the same systematic-error correction is applied consistently across data sources. - As a pre-specified item in the SAP for FDA, EMA, or HTA submissions, reported regardless of what it shows.

When NOT to use — and when it is actively misleading or dangerous

- The NCO is secretly affected by the exposure. Using an infection endpoint as the "negative" control for an immunosuppressant, a bleeding endpoint for an anticoagulant, or a fracture endpoint for a drug that alters bone density violates the null-effect assumption; a non-null result is then real, not bias, and you will wrongly discard a valid primary estimate (or wrongly "clear" it if the true effect cancels the bias). This is the dangerous failure. - Mismatched capture intensity. An outpatient "rule-out" diagnosis captured at low intensity cannot falsify an inpatient primary outcome captured at high intensity — the NCO simply has different bias, and a null NCO gives false reassurance. - Power is negligible. A rare or thinly-captured NCO yields a wide null by construction; treating that uninformative null as "the design is clean" is self-deception. Pre-specify a minimum detectable effect. - The design itself is broken. Immortal time, depletion of susceptibles, or absence of an active comparator are not rescued by a null NCO — a falsification test cannot validate a structurally invalid design. - Over-interpreting a non-null NCO. A non-null NCO proves residual bias exists; it does not identify the direction or magnitude of bias for the primary endpoint without additional assumptions (e.g., bias-transport assumptions in calibration). Never report a non-null NCO as a quantitative correction of the primary estimate. - Post-hoc cherry-picking. Choosing "cute but biologically implausible" controls after seeing the primary result, or reporting only the NCOs that came out null, converts a falsification test into a credibility theater exercise.

Data-source operational depth

- Claims (FFS or commercial): The NCO must share the primary outcome's claim type and coding opportunity — match inpatient-primary-dx to inpatient-primary-dx, not to an outpatient encounter code. The dominant failure mode is differential person-time observability: Medicare Advantage and capitated/bundled arrangements drop fee-for-service claims, so a "null" NCO can be missingness rather than absence — restrict the NCO analysis to the same Parts A/B/D (or commercial medical+pharmacy) enrolled, non-MA-only person-time used for the primary outcome. A second trap is differential competing risks: in elderly claims cohorts, if one arm has higher background mortality, a cause-specific Cox on the NCO can read as "null" while a Fine-Gray subdistribution view shows the exposure arms diverge — report the NCO on the same risk scale (cause-specific vs subdistribution) as the primary. A third trap is immortal time in procedure-anchored NCOs: anchoring an elective-procedure NCO on a post-index event reintroduces the immortal time you eliminated in the primary design. - EHR: Notes and labs help confirm an NCO is truly unrelated, but outside-care leakage (care delivered at facilities outside the system) makes the NCO look null when the bias is still present in the claims-captured primary outcome. EHR NCOs frequently encode visit frequency more than biology, so a sicker arm with more encounters accrues more NCO codes purely from contact. Define observation windows explicitly and prefer linkage to claims. - Registry: High-quality adjudication reduces misclassification of the NCO itself, but the registry population's surveillance intensity often differs from the full claims cohort, so a null NCO in the registry does not automatically clear a claims-based primary analysis. - Linked claims–EHR–registry: The ideal substrate — registry/EHR confirm the NCO's clinical unrelatedness while claims supply the utilization intensity that drives detection — but linkage selects only the linkable subset and introduces order/fill/service date discrepancies that must be reconciled before applying the identical time zero.

Worked claims example

New-user active-comparator study of Drug A vs Drug B for type 2 diabetes; primary outcome is hospitalized major bleeding (inpatient primary discharge diagnosis, validated claims algorithm). Eligibility requires 365 days of continuous, non-MA-only A/B/D enrollment before the first qualifying fill (`fill_date`), no prior fill of either drug class in that washout (incident-user), arm assigned from the NDC on `index_date`, baseline covariates measured only in [`index_date` − 365, `index_date`], 1:1 PS matching with standardized differences < 0.1, and follow-up censored at disenrollment, death, end of data, and switch. Primary result after matching: HR = 0.71 (95% CI 0.58–0.87). The pre-specified NCO is hospitalized community-acquired pneumonia — no plausible effect of either glucose-lowering drug, but captured at the same inpatient-primary-diagnosis intensity and subject to the same frailty/healthy-user selection that could drive a spurious bleeding benefit. Applying the identical matched set, weights, censoring, and `id`-based robust variance to the pneumonia NCO yields HR = 0.96 (0.83–1.11) — a null that supports removal of the shared bias. Had the pneumonia HR come back at 0.66 (0.55–0.79), the bleeding result would be presumed contaminated by residual healthy-user selection and reported with a strong caveat. The NCO is run on the same cause-specific hazard scale as the primary, restricted to the same FFS-observable person-time, and reported regardless of direction.

Interpreting the output

From the worked example: primary HR = 0.71 (95% CI 0.58–0.87) for bleeding. NCO (pneumonia, identical matched cohort and Cox specification) HR = 0.96 (95% CI 0.83–1.11) — spanning 1.0, consistent with null.

(1) Formal interpretation. The NCO estimate near 1.0 is consistent with the absence of detectable residual bias operating through shared sources — healthy-user selection, frailty channeling, inpatient intensity differences — that the pneumonia control is designed to capture. It does not prove the primary bleeding estimate is unconfounded; it only shows that the particular bias structure the NCO proxies left no detectable signal in this sample and endpoint. A non-null NCO (e.g., HR 0.66, 95% CI 0.55–0.79) would indicate bias presence and suggest its direction, but would not quantify its exact magnitude on the bleeding outcome — the two endpoints may share only some of the same confounders, and bias magnitudes can differ even when direction matches.

(2) Practical interpretation. The pneumonia NCO HR 0.96 supports proceeding with the bleeding finding: the shared confounding pathways this control can detect appear negligible. Had it returned strongly protective (e.g., ≈ 0.66), the primary HR 0.71 would be presumed contaminated and a regulator or payer reviewing the dossier would rightly treat the bleeding benefit as unestablished pending redesign or empirical calibration. The NCO result must be reported regardless of direction; selective non-reporting of unfavorable falsification tests is a protocol deviation.

Worked example

Scenario

A claims-based study compares Drug A versus Drug B for type 2 diabetes to see whether Drug A lowers the risk of hospitalized major bleeding (the primary outcome). After matching patients on measured characteristics, the analysis estimates that Drug A users have 29% lower bleeding risk (HR 0.71). The research team pre-specified appendicitis as a negative control outcome: neither drug has any known biological or behavioral pathway that would cause or prevent appendicitis, yet patients hospitalised for appendicitis would be captured in exactly the same inpatient claims records, and the same healthy-user tendencies that make Drug A users healthier overall would also affect appendicitis counts. The team runs the identical matched analysis swapping in appendicitis as the outcome.

Dataset

Summary results table — the same matched cohort, two outcome definitions. HR below 1.0 means Drug A users had fewer events.

Steps

• Choose the negative control outcome before seeing any results: appendicitis is biologically unrelated to both drugs, is captured by the same inpatient primary-diagnosis claims code type as the bleeding outcome, and would be affected by the same healthy-user selection pressures.

• Run the exact same matched analysis — same patients, same matching weights, same follow-up rules, same statistical model — changing only which outcome is the dependent variable.

• Read the negative control result first. The appendicitis HR is 0.97 (95% CI 0.60 to 1.58), which spans 1.0 comfortably — consistent with the true null effect we expect.

• Because the negative control came back null, the shared bias sources (healthy-user selection, differential care-seeking) appear to have been adequately controlled by the matching.

• The primary bleeding result of HR 0.71 is therefore more credible: the design passed its own sanity check. Had the appendicitis HR come back at, say, 0.65 (a spurious protective effect no drug could cause), you would know residual healthy-user bias is inflating the bleeding benefit too, and you would report the primary result with a strong caution.

Result

Negative control HR 0.97 (95% CI 0.60 to 1.58) — null, as required. Zero bias signal detected. The primary HR 0.71 survives the falsification check and can be reported with greater confidence. If the negative control had shown HR 0.65, that 35% spurious signal would indicate the primary estimate is contaminated by residual bias of similar or greater magnitude.

Runnable example

import numpy as np
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf
from lifelines import CoxPHFitter

def run_nco_cox(analytic: pd.DataFrame, covariates: list[str]) -> pd.Series:
    """Weighted Cox (cause-specific hazard) on the NCO, robust SE clustered on person_id.
    Mirrors the primary time-to-event specification exactly; only the outcome columns differ."""
    cols = ["arm", "iptw", "person_id", "nco_time", "nco_event"] + covariates
    cph = CoxPHFitter()
    cph.fit(analytic[cols], duration_col="nco_time", event_col="nco_event",
            weights_col="iptw", cluster_col="person_id", robust=True,
            formula="arm + " + " + ".join(covariates))
    s = cph.summary.loc["arm"]
    return pd.Series({"HR": np.exp(s["coef"]),
                      "lcl": np.exp(s["coef lower 95%"]),
                      "ucl": np.exp(s["coef upper 95%"]), "p": s["p"]})

def run_nco_rr(analytic: pd.DataFrame, covariates: list[str]) -> pd.Series:
    """Weighted log-Poisson with HC0 robust variance -> adjusted RISK RATIO on a fixed-window binary NCO.
    Poisson with log link + robust SE gives a valid RR and avoids log-binomial convergence failures."""
    formula = "nco_binary ~ arm + " + " + ".join(covariates)
    fit = smf.glm(formula, data=analytic, family=sm.families.Poisson(),
                  freq_weights=analytic["iptw"]).fit(cov_type="HC0")
    beta, se = fit.params["arm"], fit.bse["arm"]
    return pd.Series({"RR": np.exp(beta),
                      "lcl": np.exp(beta - 1.96 * se),
                      "ucl": np.exp(beta + 1.96 * se), "p": fit.pvalues["arm"]})
# A null NCO HR/RR supports the design; a non-null one warns of residual bias (direction/magnitude
# for the primary endpoint still require calibration or QBA assumptions).

library(survival)
library(sandwich)
library(lmtest)

# Weighted Cox (cause-specific hazard) on the NCO; cluster-robust SE via id-clustering.
run_nco_cox <- function(analytic, covariates) {
  f <- reformulate(c("arm", covariates), response = "Surv(nco_time, nco_event)")
  fit <- coxph(f, data = analytic, weights = iptw, cluster = person_id, robust = TRUE)
  ci <- summary(fit)$conf.int["arm", c("exp(coef)", "lower .95", "upper .95")]
  c(HR = ci[[1]], lcl = ci[[2]], ucl = ci[[3]])
}

# Weighted log-Poisson with HC0 robust variance -> adjusted RISK RATIO on a fixed-window binary NCO.
run_nco_rr <- function(analytic, covariates) {
  f <- reformulate(c("arm", covariates), response = "nco_binary")
  fit <- glm(f, data = analytic, weights = iptw, family = poisson(link = "log"))
  ct <- coeftest(fit, vcov. = vcovHC(fit, type = "HC0"))["arm", ]
  b <- ct[["Estimate"]]; se <- ct[["Std. Error"]]
  c(RR = exp(b), lcl = exp(b - 1.96 * se), ucl = exp(b + 1.96 * se))
}
# Run with the SAME weights/covariates as the primary outcome; report point estimate + CI regardless of result.