E-value Sensitivity Analysis

In plain language

The E-value asks one question about an observational result: how strong would a confounder you forgot to measure have to be to wipe out the association you found? It turns your reported risk ratio into a single number on the same risk-ratio scale, so you can say 'a hidden confounder would need to be linked to both the treatment and the outcome by at least this much to explain my result away.' A bigger E-value means the finding is harder to overturn. It is just a what-if benchmark, not proof that such a confounder exists, and it only speaks to confounding, not to other problems like selection or measurement error.

The E-value (VanderWeele & Ding, 2017) is a one-number summary of how robust an observational association is to unmeasured confounding. It is the minimum strength of association — expressed as a risk ratio — that a hypothetical unmeasured confounder would need to have both with the exposure and with the outcome, over and above the measured covariates already adjusted for, to reduce the observed effect to the null (the point-estimate E-value) or to move the confidence-interval limit nearest the null across the null (the CI E-value). It is computed entirely from the reported effect estimate and its interval; it requires no new data, no specification of which confounder is missing, and no distributional assumptions about that confounder. It rests on the Ding & VanderWeele (2016) bounding factor, which shows that the maximum factor by which confounding can inflate (or deflate) an observed risk ratio is B = (RR_AU RR_UY) / (RR_AU + RR_UY − 1), and inverts that bound to the symmetric solution E = RR_obs + sqrt(RR_obs (RR_obs − 1)) for RR_obs ≥ 1 (apply to 1/RR_obs for protective effects).

Core conceptual distinction

The E-value is a bias-analysis descriptor, not an estimator and not a bias correction. It does not change the point estimate, does not identify the missing confounder, and does not assert that such a confounder exists. It answers exactly one counterfactual question: "Given everything I already adjusted for, how strong would the worst-case remaining confounder have to be to overturn this result?" Two further distinctions matter. (1) Point-estimate vs CI E-value: the point E-value bounds the estimate, but the policy-relevant claim ("the CI no longer excludes the null") uses the CI E-value, which is always smaller and always the more honest robustness number — report both. (2) Scale of the input: the bounding factor is defined for the risk ratio. A hazard ratio for a rare outcome (or short follow-up) is used directly as an approximate RR; an odds ratio from a rare outcome is converted with the approximation RR ≈ sqrt(OR) before the bound is applied; risk differences and continuous outcomes require their own transformations. Feeding a non-rare OR or HR in as if it were an RR overstates robustness.

Pros, cons, and trade-offs

- vs negative-control outcomes/exposures: The E-value gives a quantitative bound that any reader can compute and interpret without naming a specific confounder, whereas negative controls detect the presence and direction of residual bias empirically but require a valid control variable. The E-value's cost is that it is worst-case and uninformative about whether a confounder that strong is plausible; it cannot, by itself, tell you the result is biased. Prefer the E-value as the routine, always-reportable summary; prefer negative controls when you can construct credible ones and want empirical evidence (not a bound) about residual confounding. They are complements, not substitutes. - vs full probabilistic / Monte-Carlo quantitative bias analysis (QBA, e.g. the Lash/Fox/Greenland array and bias- parameter approach): The E-value needs no bias parameters and produces one transparent number; full QBA produces a distribution of bias-adjusted estimates under explicit, defensible assumptions about the prevalence and strength of the confounder. Prefer the E-value for communicability and as a screen; prefer full QBA when stakeholders need a corrected estimate with uncertainty, or when external information about the likely confounder exists (e.g., a validation substudy or a literature RR for the suspected confounder). - vs empirical calibration: Empirical calibration uses a large set of negative-control associations to recalibrate p-values and intervals for systematic error; it is data-hungry and study-specific. The E-value is universal and distribution-free but addresses only confounding magnitude, not the full error distribution. - Critical-appraisal caveat (Sjölander, 2022; Ioannidis and others): the E-value is neither uniformly "optimistic" nor "pessimistic." Because it is reported on the RR scale and assumes a single binary confounder at its worst, it can be misread as a high bar when the true number of weak confounders acting jointly is what matters. Do not interpret a large E-value as proof of causation, nor a small one as proof of bias.

When to use

Report an E-value alongside every primary and key secondary comparative estimate from an observational RWE analysis once measured confounding has been controlled (PS matching, IPTW, overlap weighting, high-dimensional PS, or multivariable regression). It is most useful where confounding by indication is the chief residual threat, where a single unmeasured factor (frailty, disease severity, socioeconomic status, smoking) is the obvious candidate, and where decision- makers (FDA, EMA, HTA bodies) expect a transparent robustness statement. Anchor the interpretation: compare the E-value to the observed RRs of the strongest measured confounders — if the E-value is smaller than associations you already adjusted for, the result is fragile.

When NOT to use — and when it is actively misleading or dangerous

- As a substitute for design. An E-value cannot rescue a study with no active comparator, immortal time, depletion of susceptibles, or a prevalent-user cohort. Reporting a large E-value on a structurally biased estimate launders bad design into false reassurance — the single most dangerous misuse. - When the dominant threat is not confounding. The bounding factor addresses unmeasured confounding only. It says nothing about selection bias, differential outcome misclassification, immortal-time bias, or informative censoring, yet a reported E-value invites readers to believe "robustness" has been demonstrated. Never let an E-value stand in for addressing those biases directly. - On the wrong scale. Applying the RR-scale formula to a non-rare OR or a non-rare HR inflates the E-value and overstates robustness. Convert first. - When the CI already includes the null. The CI E-value is then 1 by construction — there is nothing to explain away — and reporting it as evidence of anything is meaningless. - As a threshold. There is no "E-value > X means causal" rule. Treating a cutoff as a pass/fail gate misrepresents a continuous worst-case descriptor as a hypothesis test.

Data-source operational depth

- Claims (FFS vs MA): The E-value is computed from the adjusted estimate, so its credibility inherits every limitation of the upstream claims cohort. State explicitly which confounders were measured (comorbidities via ICD, prior utilization, concomitant NDC classes) and which were not (frailty, BMI, smoking, SES, disease severity). A standing failure mode: Medicare Advantage enrollees lack fee-for-service claims, so "no prior fill" can be missingness rather than a true washout — if MA-only person-time leaks into the cohort, exposure and confounder ascertainment are differentially incomplete and the input RR is already biased, making the E-value a bound on the wrong number. Restrict to fully-observable FFS (Parts A/B/D) person-time before trusting any robustness claim. - Differential competing risks in elderly claims: In older claims populations, the comparator arms can have different mortality (a competing risk for non-fatal outcomes). If the primary estimate ignores this, the E-value bounds a cause-specific or naive estimate that is itself distorted; compute the E-value on the estimand you actually report (subdistribution vs cause-specific) and say so. - Immortal time in procedure/initiation studies: When time zero is misaligned (e.g., follow-up begins at diagnosis but exposure is a later procedure), immortal-time bias contaminates the HR. An E-value on that HR is bias analysis on top of bias — fix the time-zero structure first, then bound. - EHR: Richer measured covariates (labs, vitals, NLP-derived severity) typically reduce residual confounding and lower the E-value the analysis "needs," but visit-driven, informative-presence capture means sicker patients are seen more, creating selection the E-value does not address. Note residual unmeasured factors (unrecorded lifestyle, out-of-system care). - Registry / linked: Registries strengthen severity and adjudicated outcomes (smaller plausible residual confounding); linked claims–EHR–vital-records is the strongest substrate but introduces linkage selection. The E-value is reported the same way; what changes is how plausibly large a residual confounder could be, which is the interpretation, not the arithmetic.

Worked claims example

Question: incident heart failure with antihypertensive class A vs class B among adults with hypertension in a commercial + Medicare FFS database, using an active-comparator new-user cohort. Eligibility requires 365 days of continuous medical + pharmacy enrollment before the first qualifying fill (FFS-observable only — MA-only person-time excluded so "no prior fill" is a real washout), with the arm assigned from the NDC dispensed at time zero and `days_supply` used to build on-treatment follow-up. After 1:1 propensity-score matching on baseline covariates measured in the [index_date − 365, index_date] window (standardized differences < 0.1), the Cox model gives an adjusted HR = 0.78 (95% CI 0.65–0.93) for class A vs B. Because incident HF over the follow-up is reasonably rare, treat the HR as an approximate RR. The point-estimate E-value = 1.88 (= 1/0.78 + sqrt((1/0.78)·(1/0.78 − 1))): an unmeasured confounder would have to be associated with both treatment choice and HF by a risk ratio of at least 1.88 each, beyond the matched covariates, to fully explain the apparent benefit. The CI E-value = 1.36 for the upper limit 0.93: a confounder associated by 1.36 with both would suffice to move the interval to include the null. Now anchor it: the strongest measured confounder in the matched cohort, baseline chronic kidney disease, is associated with HF by roughly RR 1.5 — larger than the 1.36 CI E-value. The honest reading is that a single residual confounder no stronger than CKD (e.g., frailty or socioeconomic disadvantage that channels prescribing) could erase statistical significance, so the finding is only modestly robust and the manuscript should foreground negative-control outcomes and, ideally, a probabilistic bias analysis rather than rest on the E-value alone. Report both numbers (point 1.88, CI 1.36) with the list of covariates already controlled.

Interpreting the output

A study reports RR = 1.8 (95% CI 1.3–2.5). The E-value analysis yields: point-estimate E-value = 3.0; confidence-interval E-value (for the lower limit 1.3) ≈ 1.92.

(1) Formal interpretation. The point E-value of 3.0 means that an unmeasured confounder would need to be associated with both the exposure and the outcome by a risk ratio of at least 3.0 — jointly — to fully explain away the observed RR of 1.8. Any confounder whose association with either the exposure or the outcome falls below 3.0 cannot account for the entire association on its own. The CI E-value of 1.92 is the minimum joint association strength needed to shift the lower confidence limit to 1.0, rendering the result no longer distinguishable from the null at the conventional threshold. The E-value does not state the probability that confounding explains the result; it is a bounding argument about the minimum strength required for a single unmeasured binary confounder.

(2) Practical interpretation. An E-value is judged against the known confounders in the data. In the antihypertensive example (HR = 0.78, point E-value ≈ 1.88, CI E-value ≈ 1.36), the strongest measured confounder — CKD — carries an adjusted RR of approximately 1.5 with both exposure and outcome, which exceeds the CI E-value of 1.36. This means a single unmeasured confounder no stronger than CKD could erase statistical significance; the finding is only modestly robust. An E-value is one input to the overall bias assessment, not a substitute for a negative-control outcome or a full probabilistic bias analysis.

Worked example

Scenario

An analyst ran an observational cohort study and, after adjusting for every confounder they could measure, found that a drug exposure was associated with a higher risk of a rare adverse event: adjusted risk ratio RR = 1.8 with a 95% confidence interval of 1.3 to 2.5. A reviewer asks the obvious skeptic's question: 'Could a single confounder you didn't measure (say, disease severity) be faking this whole association?' The E-value answers it as one number. We compute two: one for the point estimate (1.8) and one for the confidence-interval limit nearest the null (1.3).

Dataset

The analyst does not need patient-level rows here. The E-value is computed entirely from the already-reported summary statistic and its interval, so the 'data' is one line of model output.

Steps

• The estimate is harmful (RR = 1.8, which is above 1.0), so we use the formula straight away: E = RR + sqrt(RR * (RR - 1)).

• Point-estimate E-value: plug in RR = 1.8. First the product inside the root: 1.8 (1.8 - 1) = 1.8 0.8 = 1.44. The square root of 1.44 is exactly 1.2. So E = 1.8 + 1.2 = 3.0.

• Read that out loud: an unmeasured confounder would need to be associated with BOTH the exposure AND the outcome by a risk ratio of at least 3.0 each, on top of everything already adjusted for, to pull the estimate all the way back to 1.0.

• CI E-value: the significance claim depends on the interval limit closest to 1.0, which is the lower limit 1.3 (since the estimate is harmful). Apply the same formula to 1.3: 1.3 (1.3 - 1) = 1.3 0.3 = 0.39, and sqrt(0.39) = 0.6245. So E = 1.3 + 0.6245 = 1.92.

• Interpret the pair: it takes a confounder of strength 3.0 to erase the estimate, but only 1.92 to erase statistical significance. The CI E-value is always the smaller, more honest number, so report both.

• Final gut-check: compare these to the strongest confounder you already measured. If a confounder you adjusted for was itself associated with the outcome by an RR around 2, then a residual confounder of strength 1.92 is entirely plausible, and the result is only modestly robust.

Result

Point-estimate E-value = 1.8 + sqrt(1.8 0.8) = 1.8 + sqrt(1.44) = 1.8 + 1.2 = 3.0. CI E-value (using the lower limit 1.3) = 1.3 + sqrt(1.3 0.3) = 1.3 + sqrt(0.39) = 1.3 + 0.62 = 1.92. Report both: a confounder must reach RR 3.0 with both exposure and outcome to nullify the estimate, but only RR 1.92 to make the confidence interval include the null.

Runnable example

import math

def _ev(rr: float) -> float:
    # Bounding-factor E-value for a single risk ratio; symmetric for protective effects.
    if rr < 1.0:
        rr = 1.0 / rr
    return rr + math.sqrt(rr * (rr - 1.0))

def _to_rr(est: float, scale: str, rare_outcome: bool = True) -> float:
    # Convert the reported estimate toward the risk-ratio scale the bound is defined on.
    if scale == "RR" or scale == "HR" and rare_outcome:
        return est                       # HR ~ RR when the outcome is rare / follow-up short
    if scale == "OR" and rare_outcome:
        return math.sqrt(est)            # RR ~ sqrt(OR) for a rare outcome
    raise ValueError("Provide an RR, or a rare-outcome HR/OR; otherwise convert first.")

def e_value(estimate: float, ci_low: float, ci_high: float,
            scale: str = "HR", rare_outcome: bool = True) -> dict:
    """E-values for the point estimate and the CI limit nearest the null.

    estimate/ci_low/ci_high: the adjusted effect and its 95% CI on `scale`.
    scale: 'RR', 'HR', or 'OR' (HR/OR require rare_outcome=True for the approximation).
    """
    rr  = _to_rr(estimate, scale, rare_outcome)
    lo  = _to_rr(ci_low,  scale, rare_outcome)
    hi  = _to_rr(ci_high, scale, rare_outcome)
    ev_point = _ev(rr)
    if lo <= 1.0 <= hi:                  # CI already crosses the null on the RR scale
        ev_ci = 1.0
    else:                                # use the limit closest to the null
        limit = hi if rr < 1.0 else lo
        ev_ci = _ev(limit)
    return {"rr_scale_estimate": rr, "e_value_point": ev_point, "e_value_ci": ev_ci}

# Worked claims example: PS-matched antihypertensive A vs B on incident HF (rare outcome).
# Adjusted Cox HR = 0.78 (95% CI 0.65-0.93)  ->  point E-value 1.88, CI E-value 1.36.
res = e_value(0.78, 0.65, 0.93, scale="HR", rare_outcome=True)
print(f"E-value (point) = {res['e_value_point']:.2f}")
print(f"E-value (CI)    = {res['e_value_ci']:.2f}")

library(EValue)

# Worked claims example: PS-matched antihypertensive A vs B on incident HF.
# Adjusted Cox HR = 0.78 (95% CI 0.65-0.93); incident HF is rare over follow-up.
hr_est <- 0.78; hr_lo <- 0.65; hr_hi <- 0.93

# Returns RR-scale estimate, then point-estimate and CI-limit E-values (lower/upper).
ev_hr <- evalues.HR(est = hr_est, lo = hr_lo, hi = hr_hi, rare = TRUE)
print(ev_hr)
#            point     lower    upper
# RR          0.78     0.65     0.93
# E-values    1.88     1.36       NA     <- point E-value 1.88, CI E-value 1.36

# Logistic example (rare outcome): adjusted OR and its CI from PROC/glm output.
# ev_or <- evalues.OR(est = 1.45, lo = 1.10, hi = 1.92, rare = TRUE)

# Presentation contour: combinations of RR_AU and RR_UY that would explain the estimate.
bias_plot(rr = hr_est, xmax = 4)