Selection Bias Sensitivity Analysis

In plain language

A selection bias sensitivity analysis asks: if the people who ended up in a study were not a random slice of all eligible patients, how wrong could our result be? When who gets into the study depends on both what treatment they received and how sick they were, the comparison is already tilted before any statistics are run. This analysis deliberately varies assumptions about how different the excluded patients were, then recalculates the effect estimate to find the range of plausible true answers. A small, stable range means the finding is robust; an estimate that flips direction under modest assumptions signals fragility.

Selection bias

in real-world evidence (RWE) is the bias that arises when being in the analysis — present in the database, meeting the cohort definition, surviving the continuous-enrollment requirement, being captured by EHR visits, getting linked, having complete covariates, or remaining under follow-up — is a common effect of exposure and outcome (or of their causes). In the structural language of Hernán, Hernández-Díaz, and Robins, conditioning on such a selection indicator S opens a non-causal path because S is a collider on the path exposure → S ← outcome (or exposure → S ← U → outcome). Restricting the analysis to S = 1 is exactly that conditioning, so the association you estimate is contaminated whether or not you ever fit a model term for S. A selection-bias sensitivity analysis does not pretend the data fix this; it makes the selection mechanism explicit and answers a quantitative question: given plausible assumptions about how selected and unselected patients differ, how far — and in which direction — could the reported effect move, and is there a setting that overturns the conclusion?

Two families of methods do this work. The first is the bias-parameter / tipping-point family (Greenland 1996; Lash, Fox, Fink 2014; Smith and VanderWeele 2019): posit selection-association parameters — e.g., how strongly exposure predicts selection and how strongly the outcome predicts selection among the selected-out — and either deterministically recompute the corrected estimate over a grid (a tipping-point sweep) or draw the bias parameters from prior distributions and propagate them by Monte Carlo (probabilistic bias analysis, PBA). Smith and VanderWeele give a closed-form bounding factor that yields the maximum selection bias on the risk-ratio scale for a given pair of selection associations, the selection-bias analogue of the E-value. The second family is reweighting: when the variables that drive selection are measured, fit a model for the probability of selection and analyze the selected sample with inverse-probability-of-selection weights (IPSW), the cross-sectional analogue of inverse-probability-of-censoring weighting (IPCW) used for attrition. The bias-parameter family is the only option when the selection driver is unmeasured (the usual case for who-enters-the-database); IPSW is preferred when you can credibly model selection.

Core conceptual distinction

. Selection bias is structurally distinct from confounding, and the distinction governs the fix. Confounding is a common cause of exposure and outcome and is in principle removable by adjustment, matching, or weighting on measured confounders. Selection bias is created by conditioning on a common effect (the collider S); adjusting for more baseline confounders does not remove it and can make it worse if the added covariate is itself a collider descendant. The estimand must therefore be stated as a contrast in a target population (e.g., all initiators meeting clinical eligibility), not in the selected sample — IPSW transports the selected-sample estimate back to the target, while a tipping-point analysis bounds the gap between the two. Note also which selection mechanism you are treating: cross-sectional selection-into-sample (enrollment, linkage, complete-case) is handled by IPSW or a bias-parameter sweep, whereas selection during follow-up (differential attrition) is the time-to-event analogue and is handled by IPCW with selection sensitivity layered on the censoring weights.

Pros, cons, and trade-offs

- vs simply reporting the complete-case / selected-sample estimate: the sensitivity analysis quantifies a bias the primary analysis silently assumes away. Cost: it requires explicit, debatable bias parameters or a selection model; a poorly justified prior can manufacture false reassurance or false alarm. Prefer it whenever selection plausibly depends on both exposure and outcome. - Bias-parameter / tipping-point + PBA vs IPSW: the bias-parameter approach needs no measurement of the selection driver and directly answers "how strong would selection have to be to overturn this?" — the right tool when entry into the database is governed by unmeasured factors. Cost: the parameters are assumptions, not estimates, and the bounding factor gives a worst case, not the actual bias. IPSW estimates the correction from data but is only valid if selection is captured by measured covariates (a no-unmeasured-selection assumption) and the weight model is correct; extreme weights inflate variance and can be unstable. Prefer IPSW when the selection variables are observed and rich (e.g., linked-vs-unlinked predictors); prefer the bias-parameter family when they are not, and report both when feasible. - vs the E-value / unmeasured-confounding sensitivity analysis: the E-value bounds confounding; the Smith-VanderWeele selection bounding factor is its selection-bias counterpart and answers a different question. Using an E-value to "cover" selection bias is a category error. Prefer the selection bounding factor when the threat is collider/selection structure rather than an omitted common cause. - vs IPCW alone for attrition: IPCW reweights for informative censoring under a no-unmeasured-censoring assumption; a selection sensitivity analysis stress-tests that assumption by varying outcome risk among the selected-out. Use them together — IPCW as the primary handling, selection sensitivity as the assumption check.

When to use

. Use a selection-bias sensitivity analysis whenever (a) cohort entry or retention plausibly depends on both exposure and outcome or their common causes — continuous-enrollment restrictions, mortality/registry linkage, EHR visit-based capture, complete-case covariate filters, or differential loss to follow-up; (b) a regulator or HTA reviewer will ask "what if the excluded patients differed?"; or (c) the primary analysis discards a non-trivial, potentially non-random fraction of the source population. It is mandatory rather than optional when the funnel from source to analytic cohort is steep and exposure-related.

When NOT to use — and when it is actively misleading or dangerous

- When selection cannot depend on the outcome (or its causes). If inclusion is governed only by exposure-independent administrative facts unrelated to outcome risk, there is no collider and a selection sensitivity analysis is theater — spend the effort on confounding instead. - As a substitute for fixing a fixable problem. If the selection driver is measured, the honest move is IPSW (or redefining the cohort), not a hand-tuned bias-parameter sweep chosen to land where you want. - With self-serving priors. PBA with priors quietly centered to produce a null-crossing (or a "robust" away-from-null) result is worse than no analysis: it launders an assumption as evidence. Priors must be pre-specified and defensible. - Misapplied to confounding or measurement error. The bounding factor and IPSW here address conditioning on a collider; using them to claim robustness to unmeasured confounding or outcome misclassification overstates what was tested. - When the weight model is badly misspecified or positivity fails. If some target-population strata have near-zero estimated probability of selection, IPSW weights explode and the "corrected" estimate is driven by a handful of observations — the cure is worse than the disease.

Data-source operational depth

. - Claims (FFS vs MA): The dominant selection step is the continuous-enrollment / washout requirement. In Medicare, fee-for-service (Parts A/B/D) yields complete encounter and pharmacy capture, but Medicare Advantage enrollees generate encounter records that are notoriously incomplete in some files, so "no qualifying claim" can be missingness, not a true absence of the event or fill. If MA penetration differs by the regions/employers where one treatment dominates, exclusion of MA-only person-time becomes exposure-related selection. Failure mode: requiring 365 days of continuous enrollment differentially drops the sicker arm (who churn coverage faster after a diagnosis). Workaround: report retained-vs-excluded baseline characteristics, restrict to A/B/D (or commercial medical+pharmacy) enrollees, and run a selection sweep on outcome risk among the excluded. Differential competing risks in elderly claims also drive selection: if one arm has higher early mortality, survivors selected into longer follow-up are a biased set — pair cause-specific vs subdistribution estimands with the sensitivity analysis. - EHR: Selection is visit-driven capture — a patient is "observed" only when they generate an encounter, so sicker patients are over-observed and patients who leave the system (move, switch providers, die out-of-network) are differentially lost. Complete-case filters on labs/vitals/staging compound this because missingness in EHR is itself informative (a lab is ordered because of clinical suspicion). Workaround: define observation windows explicitly, model the probability of being a complete case for IPSW, and vary outcome rates among the dropped-out in the sweep. - Registry: Participation/consent and site enrollment select on disease severity and access. Adjudicated outcomes are a strength, but the enrolled cohort may not represent the source population. Workaround: link to claims/death index to characterize non-participants where possible; otherwise bound with bias parameters. - Linked claims–EHR–vital records: Linkage itself is the selection mechanism — only the linkable subset is analyzed, and linkage probability correlates with age, insurance stability, and data completeness, all of which track outcome risk. Immortal time in procedure/linkage studies can masquerade as selection if the linkage requires surviving to a downstream record. Workaround: compare linked vs unlinked on every available variable, weight by inverse probability of linkage, and sweep outcome rates among unlinked records.

Worked claims example

Question: 1-year risk of hospitalized heart failure with drug A vs active comparator B among adults with type 2 diabetes in a commercial + Medicare FFS database, observed risk difference RD_obs = −0.04 (A lower). Cohort construction requires 365 days of continuous A/B/D (or commercial medical+pharmacy) enrollment before the first qualifying fill (`fill_date`, `days_supply`, NDC → arm), a drug-free washout for incident-user status, and ≥1 year of potential follow-up. That last requirement is a selection-into-analysis step: patients who disenroll early are dropped, and disenrollment is faster in the comparator arm (worse-controlled patients churn). Suppose 18% of the A arm and 12% of the B arm are excluded for incomplete follow-up, and you fear the excluded had higher HF risk. (1) Tipping-point sweep: let r_A and r_B be the 1-year HF risk among the excluded in each arm; the corrected RD is approximately RD_obs · 0.82·0.88 + (0.18·r_A − 0.12·r_B) reweighted to the full eligible cohort. Sweeping r_A, r_B over 0.05–0.30 shows the estimate flips sign once excluded-A risk exceeds excluded-B risk by roughly 0.08 — a clinically implausible gap, supporting robustness. (2) IPSW, if the selection driver is observed: model P(complete follow-up | baseline age, prior HF, comorbidity count, prior utilization, MA-region proxy, arm); stabilized weights sw = P(selected) / P(selected | covariates) reweight the analyzed cohort back to all eligible initiators; refit the risk model weighted and compare to RD_obs, truncating weights at the 1st/99th percentile and reporting the truncated and untruncated estimates. (3) Bounding factor: with selection-exposure association RR_AS and selection-outcome association RR_US, the Smith-VanderWeele factor BF = (RR_AS·RR_US)/(RR_AS+RR_US−1) gives the maximum factor by which the selected-sample risk ratio can differ from the target — report the BF needed to move the CI across the null.

Interpreting the output

In the Drug A versus Drug B heart-failure study, the observed risk difference = −0.04 (Arm A 8%, Arm B 12%). Selection analysis shows: Drug A — 820 included, 180 excluded (18%); Drug B — 880 included, 120 excluded (12%). The tipping grid reveals the estimate reverses sign only when excluded Drug A patients have an event rate approximately 0.22 higher than excluded Drug B patients — for example, 30% versus 8%.

(1) Formal interpretation. The sensitivity bound quantifies how extreme differential selection would need to be to explain the observed −0.04 RD. Under the Smith-VanderWeele bounding framework, both the probability of selection and the conditional outcome risk in the excluded fraction jointly determine bias magnitude. Because Drug A has proportionally more excluded patients (18% vs 12%), a scenario where those patients are sicker is plausible but must be evaluated specifically against clinical knowledge of why patients were excluded from each arm. The bounding factor also depends on the selection-exposure association; reporting the BF needed to shift the CI limit to the null anchors the assessment for a regulatory audience.

(2) Practical interpretation. A required event-rate differential of approximately 0.22 is large — roughly equal to the entire observed risk in the included Drug B group. A clinical reviewer would need to posit that excluded Drug A patients had nearly four times the event rate of excluded Drug B patients for selection bias alone to account for the protective association. Most reviewers would judge that implausible in the absence of a specific mechanism, lending moderate robustness to the −0.04 finding.

Worked example

Scenario

A claims study compares 1-year hospitalization risk for drug A versus drug B in adults with type 2 diabetes. The observed risk difference is -0.04 (drug A appears lower risk). To stay in the study, patients had to remain continuously enrolled for the full year. The analyst notices that 18% of drug A patients and 12% of drug B patients were dropped for incomplete follow-up. Sicker patients churn coverage faster, so the excluded patients probably had higher hospitalization risk than the included ones. The sensitivity analysis asks: how bad would the outcome rates in the excluded groups have to be before the conclusion reverses?

Dataset

Summary of included and excluded patients by arm, as seen in the cohort-construction log.

Steps

• The observed risk difference among included patients is 0.08 - 0.12 = -0.04 (arm A looks better).

• Let r_A be the unknown 1-year hospitalization risk among the 180 excluded arm-A patients, and r_B be the same for the 120 excluded arm-B patients.

• The corrected risk for arm A across all 1000 eligible patients is: (0.82 x 0.08) + (0.18 x r_A) = 0.0656 + 0.18 x r_A.

• The corrected risk for arm B across all 1000 eligible patients is: (0.88 x 0.12) + (0.12 x r_B) = 0.1056 + 0.12 x r_B.

• The corrected risk difference is (0.0656 + 0.18 x r_A) - (0.1056 + 0.12 x r_B), which simplifies to -0.04 + (0.18 x r_A - 0.12 x r_B).

• Vary r_A and r_B over a plausible range (0.10 to 0.30) and check whether the corrected difference crosses zero (sign flip).

Result

Tipping-point grid: the estimate flips sign only when excluded arm-A risk exceeds excluded arm-B risk by roughly 0.22 units (e.g., r_A=0.30, r_B=0.08). A gap that large is clinically implausible given baseline similarity, so the conclusion that drug A has lower risk is judged robust to this source of selection bias.

| r_A (excl. arm A) | r_B (excl. arm B) | Corrected RD | Sign flip? | |---|---|---|---| | 0.10 | 0.10 | -0.03 | No | | 0.20 | 0.10 | +0.01 | Yes | | 0.30 | 0.20 | +0.02 | Yes | | 0.10 | 0.20 | -0.05 | No |

Runnable example

import numpy as np
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf

def ipsw_risk_difference(cohort: pd.DataFrame) -> dict:
    """Reweight the SELECTED sample back to all eligible initiators via stabilized IPSW."""
    # P(selected | covariates): the selection (propensity-of-inclusion) model on the full eligible cohort.
    sel = smf.glm("selected ~ arm + age + prior_hf + comorbidity_count + prior_util + ma_region",
                  data=cohort, family=sm.families.Binomial()).fit()
    p_sel = sel.predict(cohort)
    p_marg = cohort["selected"].mean()                       # numerator of stabilized weight
    cohort = cohort.assign(sw=np.where(cohort["selected"] == 1, p_marg / p_sel, 0.0))
    # Truncate at 1st/99th pct to tame extreme weights; report both.
    lo, hi = cohort.loc[cohort.selected == 1, "sw"].quantile([0.01, 0.99])
    ana = cohort[cohort.selected == 1].copy()
    ana["sw_trunc"] = ana["sw"].clip(lo, hi)
    out = {}
    for wcol, label in [("sw", "untruncated"), ("sw_trunc", "truncated")]:
        m = smf.glm("event ~ C(arm)", data=ana, family=sm.families.Binomial(sm.families.links.Identity()),
                    freq_weights=ana[wcol]).fit()   # identity link -> coefficient IS the risk difference
        out[label] = float(m.params["C(arm)[T.B]"]) * -1   # RD for A vs B
    return out

def tipping_point(rd_obs: float, frac_excl_A: float, frac_excl_B: float,
                  grid=np.linspace(0.05, 0.30, 6)) -> pd.DataFrame:
    """Recompute the eligible-cohort RD as excluded-arm outcome risks (r_A, r_B) vary."""
    rows = []
    kept_A, kept_B = 1 - frac_excl_A, 1 - frac_excl_B
    for r_A in grid:
        for r_B in grid:
            rd_corr = rd_obs * (kept_A * kept_B) + (frac_excl_A * r_A - frac_excl_B * r_B)
            rows.append({"r_excl_A": r_A, "r_excl_B": r_B, "rd_corrected": rd_corr,
                         "sign_flip": np.sign(rd_corr) != np.sign(rd_obs)})
    return pd.DataFrame(rows)

def selection_bounding_factor(rr_select_exposure: float, rr_select_outcome: float) -> float:
    """Smith & VanderWeele (2019) max selection bias on the risk-ratio scale."""
    a, b = rr_select_exposure, rr_select_outcome
    return (a * b) / (a + b - 1.0)

library(survey)

ipsw_risk_difference <- function(cohort) {
  # Selection (probability-of-inclusion) model on the full eligible cohort.
  sel <- glm(selected ~ arm + age + prior_hf + comorbidity_count + prior_util + ma_region,
             family = binomial(), data = cohort)
  p_sel  <- predict(sel, type = "response")
  p_marg <- mean(cohort$selected)
  cohort$sw <- ifelse(cohort$selected == 1, p_marg / p_sel, 0)
  ana <- subset(cohort, selected == 1)
  q   <- quantile(ana$sw, c(0.01, 0.99))
  ana$sw_trunc <- pmin(pmax(ana$sw, q[1]), q[2])
  ana$arm_b <- as.integer(ana$arm == "B")
  rd <- function(wcol) {
    des <- svydesign(ids = ~1, weights = ana[[wcol]], data = ana)
    # Identity-link binomial -> the arm coefficient is the risk difference (A vs B = -coef).
    m <- svyglm(event ~ arm_b, design = des, family = quasibinomial(link = "identity"))
    -as.numeric(coef(m)["arm_b"])
  }
  list(untruncated = rd("sw"), truncated = rd("sw_trunc"))
}

tipping_point <- function(rd_obs, frac_excl_A, frac_excl_B, grid = seq(0.05, 0.30, 0.05)) {
  kept_A <- 1 - frac_excl_A; kept_B <- 1 - frac_excl_B
  g <- expand.grid(r_excl_A = grid, r_excl_B = grid)
  g$rd_corrected <- rd_obs * (kept_A * kept_B) + (frac_excl_A * g$r_excl_A - frac_excl_B * g$r_excl_B)
  g$sign_flip <- sign(g$rd_corrected) != sign(rd_obs)
  g
}

selection_bounding_factor <- function(rr_select_exposure, rr_select_outcome) {
  a <- rr_select_exposure; b <- rr_select_outcome
  (a * b) / (a + b - 1)
}