Quantitative Bias Analysis Toolkit

In plain language

Quantitative bias analysis (QBA) replaces the vague disclaimer that 'unmeasured confounding may affect results' with actual arithmetic: you state what you think the bias is, plug in numbers for how strong it would have to be, and report what the effect estimate would look like if that bias were real. The toolkit has three main moves — bounding (the E-value asks how strong an unmeasured factor would have to be to explain your finding away), correcting (you adjust the observed estimate using known error rates for your outcome measure), and probabilistic simulation (you let the bias parameters vary across a plausible range and see the spread of corrected results). Done honestly, QBA tells a regulator or payer how much hidden noise would need to exist before the conclusion flips.

Quantitative bias analysis (QBA)

replaces the narrative "limitations" paragraph with arithmetic: rather than asserting that residual confounding or endpoint misclassification "may bias results," QBA states a bias model, assigns values (or distributions) to its parameters, and reports the estimate you would obtain if that bias were real. In RWE and pharmacoepidemiology it is the discipline that turns an observed hazard ratio into a bias-adjusted hazard ratio, or that reports the magnitude of unmeasured confounding required to move a confidence interval across the null. It is best treated not as one procedure but as a coordinated toolkit spanning three bias families — confounding, information (misclassification), and selection — plus the empirical probes (negative controls, calibration) and anchors (validation substudies) that make the bias parameters credible.

Core conceptual distinction

Two axes organize the toolkit and are routinely conflated. (1) Deterministic vs probabilistic: a deterministic (simple) bias analysis plugs in fixed best-guess bias parameters and returns a single corrected estimate (and optionally a few scenarios); a probabilistic bias analysis (PBA) assigns prior distributions to the bias parameters, draws from them by Monte Carlo, and returns a simulation interval that propagates bias uncertainty alongside random error. (2) Bounding vs correcting: bounding methods (the E-value, Ding–VanderWeele) report how strong an unmeasured confounder must be to explain away the result without claiming to know it exists, whereas correcting methods (record-level adjustment, the Fox–Lash misclassification matrix, external/indirect adjustment) shift the point estimate to where it would sit if the named bias held. A bound answers "how robust is this?"; a correction answers "what is the answer after accounting for this?" The two are complementary, not interchangeable — a small E-value does not mean the corrected estimate is null, and a reassuring correction under one set of priors does not bound all bias mechanisms. QBA also distinguishes the target: the same machinery corrects a risk ratio, an odds ratio, or a rate ratio, but the bias formulas (and the assumption of non-differential vs differential error) differ, and the corrected estimate inherits whatever estimand the primary analysis defined.

Pros, cons, and trade-offs

- vs a single E-value (e-value-sensitivity-analysis): The full toolkit covers misclassification and selection — not just unmeasured confounding — and can correct the estimate using validation data or simulation rather than only bounding it. Cost: it is far more assumption-laden and harder to communicate than one transparent number; reviewers can dispute every prior. Prefer the E-value as the universally reportable confounding sensitivity metric and the entry point; escalate to PBA when the decision is high-stakes, when bias is differential, or when validation data exist to anchor priors. - vs empirical probes (negative-control-outcomes-rwe, empirical-calibration-negative-controls-rwe): Negative controls and empirical calibration detect and recalibrate residual systematic error using observed data, so they need no bias-parameter priors. QBA quantifies and corrects an assumed, named bias mechanism. Cost: QBA is only as credible as its priors; negative controls cannot tell you which bias is operating, only that something is. Use both — controls to demonstrate bias is present/absent, QBA to characterize its consequences. - vs deterministic-only sensitivity tables: PBA integrates over plausible bias-parameter ranges instead of reporting a handful of corner scenarios, which avoids the false precision of a single corrected number and the cherry-picking risk of "best/worst case" tables. Cost: PBA hides assumptions inside priors that reviewers may not scrutinize, and a confidently wrong prior produces a confidently wrong interval. Prefer deterministic for a transparent first pass and to expose the bias formula; prefer PBA when bias-parameter uncertainty is itself material to the decision.

When to use

Any regulatory- or HTA-grade RWE study where a residual-bias objection is foreseeable and would change the decision: confirmatory comparative safety/effectiveness studies, external-control-arm submissions, and label-expansion or coverage dossiers. Pre-specify the bias mechanisms in the SAP (not the manuscript discussion): name the suspected unmeasured confounder, the outcome-algorithm error model, and the selection/attrition mechanism, and state the priors and their sources (validation chart review, published PPV/sensitivity, expert elicitation, negative-control calibration). QBA is most valuable precisely when the primary estimate is decision-relevant but the design cannot fully exclude a specific, articulable bias.

When NOT to use — and when it is actively misleading or dangerous

- As a credibility laundromat. Reporting one favorable corrected estimate and calling the result "robust to bias" is worse than no QBA: it manufactures false reassurance. A QBA that only ever moves the estimate in the convenient direction, or that omits the bias mechanism most threatening to the conclusion, is advocacy, not analysis. - When the priors are invented. PBA with priors pulled from thin air launders guesswork into a precise-looking simulation interval. If no validation data, literature, or defensible elicitation exist for a bias parameter, say so and bound (E-value) rather than correct — a correction implies knowledge you do not have. - When the bias is structural, not parametric. QBA corrects bias whose direction and mechanism you can model. It cannot rescue a fatally non-comparable design (e.g., immortal-time bias from misaligned time zero, or a comparator prescribed to systematically different patients). Fix the design; do not "adjust it away" post hoc. - When applied to the wrong scale or with the wrong differentiality assumption. Assuming non-differential outcome misclassification when error depends on exposure (e.g., the treated arm is surveilled more intensely) can correct the estimate in the wrong direction and make a biased result look unbiased. Non-differential misclassification biases toward the null on average for a dichotomous exposure/outcome, but this is not guaranteed for polytomous variables or any single study realization — never use it as a reflexive "conservative" defense.

Data-source operational depth

- Claims (FFS vs Medicare Advantage): The dominant biases are unmeasured frailty/SES/over-the-counter and cash-pay drug use, outcome-algorithm error (claims-based endpoint PPV is often 0.6–0.9), and exposure capture. MA-only person-time lacks fee-for-service claims, so a person can appear to have "no event" or "no fill" purely because their utilization is capitated and unobserved — model this as differential missingness by plan type, not as a true negative, and prefer restricting to enrollees with complete A/B/D. Competing risks bias QBA in the elderly: if death (a competing event) is differentially captured by exposure arm, both the observed rate and any misclassification correction inherit that distortion — link to the death index before correcting outcome rates. Anchor outcome PPV/sensitivity to a chart-validated subset; absent that, use published validation estimates and widen the priors to reflect transportability uncertainty. - EHR: Endpoint phenotyping error is the headline bias and is frequently differential — sicker or more-monitored patients generate more notes, labs, and codes, inflating sensitivity in one arm. Labs are often missing-not-at-random (a test is ordered because the clinician suspected the outcome), so a complete-case analysis is a selection mechanism that QBA must model, not a benign reduction in n. External-care leakage (events treated outside the network) imposes differential outcome under-capture by patient mobility. Use NLP-validated or chart-review subsets to estimate the sensitivity/specificity matrix; allow it to differ by arm. - Registry: Strengths are adjudicated outcomes and disease severity; the QBA targets are linkage error (to claims for exposure or to death indices for censoring), registry completeness/enrollment selection, and transportability from the registry-eligible population to the broader treated population. Model the linkage match rate as a selection probability and test whether it differs by exposure. - Linked claims–EHR–vital records: The richest substrate for anchoring priors (validation against EHR/charts, real mortality), but linkage itself selects the linkable subset and creates date-discrepancy (order vs fill vs service) problems that can manufacture immortal time before any QBA is run — reconcile time zero first, then correct residual parametric bias.

Worked claims example

A commercial + Medicare FFS active-comparator new-user study reports HR = 0.78 (95% CI 0.66–0.92) for incident MI with Drug A vs Drug B after high-dimensional PS weighting; the primary cohort required 365 days of continuous A/B/D enrollment (no MA-only person-time), a drug-free washout (no prior `fill_date` of either NDC class in the lookback), index_date = first qualifying fill, and a claims MI algorithm (inpatient dx in the first position) with chart-validated PPV = 0.90 and sensitivity = 0.75. The pre-specified QBA package: (1) Bound — the E-value for the point estimate is 1.87 and for the upper CI limit is 1.39, i.e., an unmeasured confounder associated with both Drug A use and MI by a risk ratio of ~1.4, beyond the measured covariates, could move the interval to the null. (2) Confounding correction (PBA) — assume an unmeasured frailty factor with prevalence 0.25 in the Drug A arm and 0.15 in the comparator (Beta priors anchored to a validation chart-review subset) and a frailty–MI risk ratio drawn from Triangular(1.0, 1.6, 2.5); Monte Carlo over 10,000 draws yields a median bias-adjusted HR of ~0.86 with a 95% simulation interval that still excludes 1.0, so confounding of the plausible magnitude does not overturn the finding. (3) Outcome misclassification — under non-differential error (sensitivity 0.75, specificity 0.99 from validation), the corrected RR is essentially unchanged (~0.79); but if the more intensively monitored Drug A arm has sensitivity 0.85 vs 0.70 in the comparator (differential), the corrected RR attenuates toward ~0.88, the key vulnerability to flag. (4) Selection — if disenrolling treated patients carry 1.8× the MI risk and are lost 10% more often (an MA-switch mechanism), an inverse-probability-of-censoring reweight shifts the estimate modestly upward. (5) Tipping point — the conclusion (a protective effect) survives unless differential outcome sensitivity exceeds a ~15-point arm gap and an unmeasured RR_UY > 2.0 act jointly. The QBA is reported as the joint envelope of these analyses, pre-specified in the SAP, with every prior traced to validation data, published estimates, or documented elicitation — not as a single convenient corrected number.

Interpreting the output

The worked example yields a joint bias envelope: E-value 1.87 (CI bound 1.39); PBA median bias-adjusted HR ≈ 0.86 with a 95% simulation interval still excluding 1.0; non-differential misclassification correction HR ≈ 0.79; differential misclassification scenario HR ≈ 0.88.

(1) Formal interpretation. Each component answers a different question. The E-value bounds the minimum confounder strength needed to explain the result — it asserts nothing about actual confounder prevalence. The PBA simulation interval is not a confidence interval: it is an uncertainty band conditional on the analyst's chosen prior distributions for bias parameters (frailty prevalence Beta priors, RR Triangular draws); changing those priors changes the interval. The deterministic misclassification corrections are each conditional on the fixed sensitivity and specificity values supplied from chart-review validation. Collectively the envelope characterizes corrected estimates across plausible bias structures, not a single authoritative adjusted value.

(2) Practical interpretation. The protective HR 0.78 survives all individually plausible biases in this example: the PBA median remains well below 1.0 and the tipping point requires both a 15-percentage-point sensitivity gap and an unmeasured confounder RR > 2.0 acting simultaneously. The operationally realistic vulnerability is differential outcome ascertainment, which pushes the estimate toward ≈ 0.88 — narrowing but not erasing the apparent benefit. Report the full envelope pre-specified in the SAP; citing only the most favorable corrected number defeats the purpose of quantitative bias analysis.

Worked example

Scenario

A new drug (Drug A) is compared against an older drug (Drug B) for heart attack risk in a claims database. After adjusting for all measured risk factors, the observed hazard ratio is 0.78 (95% CI 0.66 to 0.92), suggesting Drug A is protective. The study team runs a three-part QBA toolkit to test how robust that finding is: first they apply the E-value bound, then a simple deterministic correction for outcome misclassification, and finally a quick probabilistic check for unmeasured confounding.

Dataset

Summary numbers entering the QBA — the observed estimate and the outcome-algorithm accuracy from chart validation

Steps

• Step 1 — E-value: The formula for the E-value of a hazard ratio uses RR* = HR + sqrt(HR × (HR - 1)). For HR = 0.78 (a protective direction, so we first flip to 1/0.78 = 1.28): E-value = 1.28 + sqrt(1.28 × 0.28) ≈ 1.28 + 0.60 ≈ 1.88 (rounded to 1.87 in the source file). For the CI limit nearest the null, HR = 0.92 -> 1/0.92 = 1.09: E-value = 1.09 + sqrt(1.09 × 0.09) ≈ 1.09 + 0.31 ≈ 1.40 (file: 1.39). This means an unmeasured factor would have to be associated with both drug choice and MI by a risk ratio of at least 1.4 — beyond all measured covariates — to move the confidence interval to the null.

• Step 2 — Deterministic misclassification correction: Using the standard non-differential correction formula with sensitivity = 0.75 and specificity = 0.99, the corrected risk ratio is very close to the observed 0.78 (approximately 0.79). The high specificity (0.99) means very few false positives contaminate the outcome count, so non-differential misclassification barely changes the estimate. The key vulnerability is differential error: if the Drug A arm is monitored more closely, its sensitivity might be 0.85 versus 0.70 in the Drug B arm. Plugging in those arm-specific values shifts the corrected HR upward to approximately 0.88 — still protective but weaker.

• Step 3 — Probabilistic bias analysis for confounding: Assume an unmeasured frailty factor is present in about 25% of Drug A users but only 15% of Drug B users (anchored to a chart-review substudy), and that this factor raises MI risk somewhere between 1.0 and 2.5-fold (most likely around 1.6). Running 10,000 Monte Carlo draws through the bias correction formula yields a median corrected HR of approximately 0.86. The 95% simulation interval still excludes 1.0, meaning the protective signal survives across the full plausible range of this confounder.

• Step 4 — Tipping point: The conclusion flips only if differential outcome sensitivity exceeds a 15-percentage-point gap between arms AND an unmeasured confounder with risk ratio above 2.0 acts at the same time. Neither condition alone is sufficient to overturn the finding.

Result

E-value for point estimate = 1.87 (CI limit E-value = 1.39): a confounder weaker than RR 1.4 cannot explain the finding away. Non-differential misclassification correction leaves HR essentially at 0.79; differential monitoring shifts it to ~0.88, the main vulnerability to flag. Probabilistic confounding analysis: median corrected HR ~0.86, simulation interval still excludes 1.0. Tipping point: finding flips only under joint worst-case bias (differential sensitivity gap >15 points AND confounder RR >2.0). The protective effect is robust to plausible individual biases but would require careful monitoring if both act simultaneously.

Methods Table

The three QBA tools applied and what each one answers

Runnable example

import numpy as np

def pba_unmeasured_confounder(observed_rr: float,
                              log_se: float,
                              n_iter: int = 100_000,
                              seed: int = 1) -> dict:
    rng = np.random.default_rng(seed)

    # Bias-parameter priors (ANCHOR these to validation data, not guesses):
    #   p_u1, p_u0 : prevalence of the unmeasured confounder in treated / comparator arms
    #   rr_uy      : confounder -> outcome risk ratio
    p_u1  = rng.beta(25, 75, n_iter)                 # ~0.25 prevalence in treated
    p_u0  = rng.beta(15, 85, n_iter)                 # ~0.15 prevalence in comparator
    rr_uy = rng.triangular(1.0, 1.6, 2.5, n_iter)    # confounder-outcome RR

    # Deterministic confounding bias factor (Bross / Schlesselman form).
    bias_factor = ((p_u1 * (rr_uy - 1) + 1) /
                   (p_u0 * (rr_uy - 1) + 1))
    corrected_rr = observed_rr / bias_factor

    # Add random error to each draw so the interval reflects bias + sampling uncertainty.
    log_corrected = np.log(corrected_rr) + rng.normal(0.0, log_se, n_iter)
    corrected_total = np.exp(log_corrected)

    return {
        "median": float(np.percentile(corrected_total, 50)),
        "ci_95": (float(np.percentile(corrected_total, 2.5)),
                  float(np.percentile(corrected_total, 97.5))),
        "p_crosses_null": float(np.mean(corrected_total > 1.0)),
    }

pba_misclass_rr <- function(a, b, c, d,
                            se_alpha, se_beta,   # sensitivity Beta prior
                            sp_alpha, sp_beta,   # specificity Beta prior
                            n = 100000L, seed = 1L) {
  set.seed(seed)
  se <- rbeta(n, se_alpha, se_beta)              # outcome sensitivity
  sp <- rbeta(n, sp_alpha, sp_beta)              # outcome specificity

  n1 <- a + b; n0 <- c + d                       # exposed / unexposed totals
  # Back-correct observed cases to expected true cases: A_true = (A_obs - (1-Sp)*N) / (Se - (1-Sp))
  a_corr <- (a - (1 - sp) * n1) / (se - (1 - sp))
  c_corr <- (c - (1 - sp) * n0) / (se - (1 - sp))

  valid <- a_corr > 0 & c_corr > 0 & a_corr < n1 & c_corr < n0
  rr <- (a_corr[valid] / n1) / (c_corr[valid] / n0)

  # Total error: add sampling variability of log(RR) from the corrected cell counts.
  log_se <- sqrt(1/a_corr[valid] - 1/n1 + 1/c_corr[valid] - 1/n0)
  rr_total <- exp(log(rr) + rnorm(sum(valid), 0, log_se))

  list(median = median(rr_total),
       ci_95  = quantile(rr_total, c(0.025, 0.975)),
       prop_kept = mean(valid))
}