Comparative Effectiveness Research (CER) Methods

In plain language

Comparative effectiveness research (CER) asks which of two treatments works better in the real world — not under idealized trial conditions but among actual patients who may skip doses, have multiple illnesses, and stay on therapy for years. Instead of randomly assigning patients to drugs, researchers build a carefully designed study from insurance claims or medical records, using statistical tools to make the two groups as comparable as possible before measuring outcomes. The core insight is that comparing a drug to an active alternative patients are already receiving is far more answerable than asking whether the drug beats doing nothing.

Comparative effectiveness research (CER) methods

are the set of design and analytic tools used to recover a comparative causal estimand — the effect of treatment strategy A versus a clinically reasonable alternative B for the same decision — from real-world data when a head-to-head randomized trial cannot be run. CER is not a single estimator; it is the methods stack that sits above the specific designs and models in this catalog. This entry is the umbrella for the whole methodological enterprise (when to reach for a propensity score versus an instrument versus a g-method versus a natural experiment, and how to defend the answer for a clinical, regulatory, or HTA decision). Its sibling entry `cer-observational` is narrower: it catalogs the observational study designs (ACNU cohort, prevalent-new-user, nested case-control, self-controlled designs) and how to pick among them. Read this entry for the analytic and inferential layer; read `cer-observational` for the design-selection layer.

Core conceptual distinction

(the core estimand distinction). Three distinctions organize all of CER and must be settled in the protocol before any code runs. (1) Efficacy vs effectiveness: an RCT estimates efficacy under protocol-enforced adherence in a selected population; CER estimates effectiveness — the effect of a strategy as actually used in routine care, with imperfect adherence, broad comorbidity, and longer horizons. These are different estimands, not noisy versions of the same number. (2) Comparative vs absolute: CER compares A vs B for one decision, because a drug-vs-nothing contrast in secondary data is almost always fatally confounded by indication; the active comparator converts an unanswerable absolute question into an answerable comparative one. (3) The causal estimand itself must be named: ATE vs ATT/ATU (which the weighting or matching scheme implicitly targets — IPTW→ATE, 1:1 matching→ATT, overlap weights→ATO), and ITT-like (initiation/intention-to-treat) vs per-protocol/as-treated (which requires censoring at switch/discontinuation and inverse-probability-of-censoring weighting to handle informative censoring). Different choices imply different models and different decision-relevance; conflating them is the most common silent error in applied CER.

The method-selection logic (the heart of CER)

The driving question is what is the dominant threat to exchangeability, and is it measured? - Confounding is measured → design out what you can (active comparator + new-user), then balance on measured covariates with a propensity score (matching, IPTW, overlap weighting, stratification) or a doubly robust estimator (PS + outcome model, e.g., AIPW/TMLE) that is consistent if either model is correct. With high-dimensional claims/EHR proxies, use a high-dimensional propensity score to recover information from variables you did not pre-specify. - Confounding is time-varying and affected by prior treatment (a confounder that is also a mediator) → standard regression and baseline PS are biased in both directions; use g-methods (marginal structural models via IPTW, g-estimation of structural nested models, or g-computation) within a target-trial frame, typically with clone-censor-weight for sustained strategies. - Unmeasured confounding is the dominant threat and a valid instrument exists (physician/regional preference, formulary rules, distance) → instrumental variables / 2SLS / 2SRI, accepting that the estimand is a local effect among compliers, not the population ATE, and that exclusion-restriction violations are untestable. - A policy or natural experiment created quasi-random variation in exposure → difference-in-differences (parallel trends), regression discontinuity (a sharp eligibility threshold), or synthetic control for a single treated unit. Whatever the route, CER discipline requires pre-specification of a target trial, transparent reporting (STaRT-RWE, HARPER, the ISPOR-ISPE good-practice recommendations), fit-for-purpose data assessment, and quantitative sensitivity analysis (E-value, negative controls / empirical calibration, quantitative bias analysis) because residual confounding is never excluded by design alone.

Pros, cons, and trade-offs

- vs the head-to-head RCT it emulates: CER delivers larger, more representative populations, rare and long-term outcomes, subgroups, and answers far faster and cheaper, and it captures real-world adherence and heterogeneity that efficacy trials suppress. Cost: residual confounding (measured and unmeasured) is never excluded, the answer depends on data quality and correct estimand specification, and regulators still treat a single observational CER study as weaker than a confirmatory RCT. The RCT-DUPLICATE program showed observational CER can reproduce RCT effect estimates when design emulation is rigorous and confounders are well measured — and fails predictably when they are not. Prefer CER when an RCT is infeasible/unethical, generalizability is the question, or speed is decisive; combine with RCT evidence via transportability or network meta-analysis when possible. - vs naive observational comparisons (prevalent-user, "ever vs never", baseline regression only): the CER stack (active comparator, new-user, PS/g-methods, target trial) removes confounding by indication, immortal time, and depletion-of-susceptibles that cripple naive analyses, and makes the estimate defensible. Cost: more complex, more assumption-heavy, and slower — and still wrong if the assumptions fail, which is exactly why sensitivity analysis is mandatory, not optional. Prefer the CER stack for essentially all non-randomized comparative questions. - PS vs g-methods vs IV (within the stack): PS methods are simplest and best when key confounders are measured and not time-varying; g-methods are necessary (not merely better) when a covariate is both confounder and mediator of a sustained strategy; IV is the only route to unmeasured-confounding identification but pays for it with a fragile, untestable exclusion restriction and a compliers-only estimand. Reaching for IV when a good PS would do trades away precision and interpretability; reaching for a baseline PS on a time-varying problem trades away validity.

When to use

Reach for the CER stack whenever the decision is comparative (strategy A vs a clinically reasonable alternative B for the same indication and the same decision point), a head-to-head RCT is infeasible, unethical, too slow, or too narrow, and a defensible active comparator exists so the question is not secretly drug-vs-nothing. Within the stack, the routing rule is mechanical: if the dominant confounders are measured and not time-varying, use a propensity score or a doubly robust estimator (add hd-PS for claims/EHR proxies); if a confounder is also a mediator of a sustained strategy, use g-methods inside a target-trial frame; if unmeasured confounding dominates and a valid instrument exists, use IV; if a policy or natural experiment created quasi-random variation, use DiD/RD/synthetic control. Use it for regulatory- or HTA-grade comparative safety/effectiveness and value questions in claims, EHR, registry, or linked data, and pre-specify the target trial and the sensitivity analyses before any code runs.

When NOT to use — and when it is actively misleading or dangerous

- No clinically interchangeable comparator exists. Forcing a comparator prescribed to systematically different patients (a second-line agent against a first-line agent) re-introduces confounding by indication and channeling — the bias you came to remove. If the honest question is drug-vs-no-treatment, the active-comparator machinery cannot answer it. - The dominant confounder is unmeasured and no valid instrument exists. E-value or a strong negative-control association that wipes out the effect is a signal to not report a causal estimate; report the bias-analysis bound instead. - Severe positivity violation. If one drug is reserved for the sickest patients, PS distributions separate, matching discards most of the cohort, weights explode, and the surviving estimand maps to no clinically meaningful population. - Time-zero / immortal-time misalignment. Starting follow-up at diagnosis rather than initiation, or selecting on a post-baseline event, manufactures bias before any model is fit; no downstream adjustment repairs it. - g-methods on thin longitudinal data. MSMs and clone-censor-weight demand correctly modeled time-varying processes and positivity at every time point; on sparse measurement they substitute model dependence for the bias they removed.

Data-source operational depth

- Claims (FFS vs MA vs commercial): the workhorse for comparative drug safety/effectiveness — large N, longitudinal fills (NDC + `fill_date` + `days_supply`), diagnoses, procedures, and costs. Strengths: near-complete utilization and pharmacy in fee-for-service. Failure modes and workarounds: Medicare Advantage encounter data are incomplete — MA-only person-time lacks reliable FFS claims, so "no prior fill" can be missingness rather than a true washout; restrict to enrollees with full Parts A/B/D (or a commercial medical+pharmacy benefit) and exclude MA-only person-time. Coding drift (ICD-9→ICD-10, fee-schedule changes) breaks longitudinal outcome algorithms — version your code lists. No labs, no severity, no reason for treatment, so channeling is invisible without linkage. Differential competing risks by exposure in elderly claims (e.g., a comparator preferentially used in frailer patients raises competing mortality) biases naive cause-specific or Kaplan-Meier outcome risk; model competing risks explicitly. 90-day mail order, sample fills, and stockpiling distort `days_supply` and on-treatment windows. - EHR: richer covariates (labs, vitals, problem lists, notes/NLP for severity and SDoH) make effect-modification and indication confirmation possible. But initiation is the order/administration, not the dispensing (link to pharmacy to confirm the patient started), capture is visit-driven so a patient who leaves the system is differentially lost (treat loss to follow-up as potentially informative), and care outside the network is invisible. - Registry: strongest for indication, disease severity, and adjudicated outcomes (cancer stage, device endpoints); typically weak for full pharmacy exposure and complete utilization. Link to claims for fills/costs and to a death index for censoring; excellent as the eligibility/severity backbone of a target-trial emulation. - Linked claims–EHR–vital records: the ideal substrate (EHR severity + claims completeness + reliable mortality) but linkage introduces selection (only the linkable subset) and order/fill/service date discrepancies that must be reconciled before time-zero assignment.

Worked claims example (head-to-head, FFS logic)

Question: 2-year risk of hospitalized heart failure with a second-generation sulfonylurea vs a DPP-4 inhibitor among adults with type 2 diabetes, in a commercial + Medicare FFS database. (1) Eligibility: age ≥18, ≥2 outpatient or ≥1 inpatient diabetes diagnosis, and 365 days of continuous medical + pharmacy enrollment (full A/B/D or commercial) before the first study fill — exclude any MA-only person-time so "no prior fill" is observed, not missing. (2) Washout (new-user): no fill of any sulfonylurea or DPP-4 inhibitor in the 365-day lookback, making both arms incident users. (3) Time zero: the date of the first qualifying fill; assign the arm from the NDC dispensed that day. (4) Covariates: measured only in the 365-day window up to and including time zero (comorbidities, prior insulin, baseline HCRU, HbA1c proxies), feeding a high-dimensional propensity score. (5) Estimand and estimator: target the ATE with stabilized IPTW and add an outcome model for double robustness (AIPW), or target the ATT with 1:1 PS matching — name the choice in the SAP. (6) Follow-up and competing risks: from time zero to first validated HF hospitalization, censoring at disenrollment, end of data, and (as-treated) discontinuation (last `days_supply` end + a pre-specified grace period) or switch; treat all-cause death as a competing risk (it is differential by arm in this older cohort) and report cause-specific HR alongside the Fine-Gray subdistribution effect, since they answer different questions. (7) Diagnostics and sensitivity: standardized mean differences <0.1 after weighting/matching, an E-value for the point estimate and confidence limit, a negative-control outcome to detect residual confounding, and sensitivity to washout length and grace period.

Interpreting the output

In the new-user active-comparator design comparing Drug A and Drug B (2-year commercial claims), propensity-score balancing produces: Drug A event rate 4.2 per 100 person-years; Drug B 6.8 per 100; risk ratio ≈ 0.62; absolute risk reduction ≈ 2.6 percentage points over 2 years.

(1) Formal interpretation. The RR ≈ 0.62 is an effectiveness estimate in the real-world initiator population — patients who received prescriptions in routine care, with their actual contraindications, prior treatment failures, and adherence patterns intact. It is not an efficacy estimate from a trial's per-protocol population. The new-user restriction starts the comparison at first dispensing, eliminating prevalent-user depletion-of-susceptibles bias. The active-comparator restriction means both arms faced a similar prescribing threshold, attenuating healthy-user confounding that would inflate apparent benefit if the comparator were untreated patients. Residual confounding by indication — Drug A may be preferred in lower-risk patients — remains the primary threat to validity and should be quantified via E-value or quantitative bias analysis, alongside a negative-control outcome assessment.

(2) Practical interpretation. A 2.6 pp absolute reduction over 2 years corresponds to approximately 26 fewer events per 1,000 treated patients. This is an actionable number for coverage decisions, but carries an effectiveness caveat: the observed benefit includes the contribution of partial adherence, switching, and provider management decisions that would not replicate under controlled trial conditions. Pair the RR with a cause-specific and subdistribution hazard ratio if death is a competing event in the population.

Worked example

Scenario

A payer wants to know whether Drug A (a newer diabetes pill) leads to fewer hospitalizations for heart failure than Drug B (an older diabetes pill) among adults who are just starting one of these two drugs. No head-to-head trial has been run. Researchers build an active-comparator, new-user study from two years of commercial insurance claims.

Dataset

Five design decisions a CER analyst documents before running any code — one row per choice, contrasting effectiveness in the real world vs what an efficacy trial would do.

Steps

• Step 1 — Define the question precisely: Drug A versus Drug B, in new users only (no one already on either drug in the prior 12 months), for adults aged 18 and older with a documented diabetes diagnosis.

• Step 2 — Assign each patient a start date (their first fill of either drug) and record all 40+ baseline characteristics measured in the 12 months before that start date.

• Step 3 — Calculate a propensity score for each patient: the predicted probability of having received Drug A given their age, sex, prior diagnoses, other medications, and health-care use.

• Step 4 — Use those scores to make the Drug A and Drug B groups statistically comparable, so confounding by indication is removed — the same way random assignment would in a trial.

• Step 5 — Follow all patients from their start date until they are hospitalized for heart failure, leave the insurance plan, or reach 730 days (2 years), whichever comes first.

• Step 6 — Compare the 2-year heart failure hospitalization rate between the two balanced groups and report the result as the real-world effectiveness difference.

Result

After propensity-score balancing, the Drug A group and Drug B group look similar on all measured baseline characteristics. The 2-year heart failure hospitalization rate is 4.2 per 100 patients in the Drug A arm versus 6.8 per 100 in the Drug B arm — a 2.6 percentage-point absolute reduction (relative risk 0.62). Because this is measured in routine care with real-world adherence and a broad patient population, it is an effectiveness estimate, not the efficacy estimate a placebo-controlled trial would produce.

Runnable example

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression

def aipw_ate(df: pd.DataFrame, covs: list[str], horizon_days: int = 730) -> dict:
    """Doubly robust ATE (risk difference) for a binary outcome by `horizon_days`.

    AIPW is consistent if EITHER the propensity model OR the outcome model is correct.
    Death-without-event is treated as a censoring competing event: subjects who die
    before the horizon without the outcome and with fu_time < horizon are censored, so
    the contrast is the cause-specific (treatment-as-cause) risk difference. For a
    policy-relevant cumulative-incidence (subdistribution) contrast, switch to a
    Fine-Gray / Aalen-Johansen estimator (see the R and SAS blocks).
    """
    d = df.copy()
    # Outcome observed by the horizon (event before horizon AND followed long enough).
    d["y"] = ((d["event"] == 1) & (d["fu_time"] <= horizon_days)).astype(int)

    X = d[covs].to_numpy()
    a = d["treat"].to_numpy()
    y = d["y"].to_numpy()

    # --- Propensity model -> stabilized IPTW ---
    ps = LogisticRegression(max_iter=1000).fit(X, a).predict_proba(X)[:, 1]
    ps = np.clip(ps, 0.01, 0.99)                 # bound to tame extreme weights
    p_treat = a.mean()
    sw = np.where(a == 1, p_treat / ps, (1 - p_treat) / (1 - ps))  # stabilized weights

    # --- Outcome model (fit within each arm to allow effect modification) ---
    mu1 = LogisticRegression(max_iter=1000).fit(X[a == 1], y[a == 1]).predict_proba(X)[:, 1]
    mu0 = LogisticRegression(max_iter=1000).fit(X[a == 0], y[a == 0]).predict_proba(X)[:, 1]

    # --- AIPW influence-function estimates of E[Y^1] and E[Y^0] ---
    psi1 = mu1 + (a / ps) * (y - mu1)
    psi0 = mu0 + ((1 - a) / (1 - ps)) * (y - mu0)
    rd = psi1.mean() - psi0.mean()
    se = np.sqrt(np.var(psi1 - psi0, ddof=1) / len(d))  # influence-function SE

    # --- Weighted covariate balance (standardized mean differences) ---
    def wsmd(col):
        t, c = d[a == 1], d[a == 0]
        wt, wc = sw[a == 1], sw[a == 0]
        mt = np.average(t[col], weights=wt); mc = np.average(c[col], weights=wc)
        vt = np.average((t[col] - mt) ** 2, weights=wt)
        vc = np.average((c[col] - mc) ** 2, weights=wc)
        return (mt - mc) / np.sqrt((vt + vc) / 2 + 1e-12)

    smd = {c: round(float(wsmd(c)), 3) for c in covs}
    return {"risk_diff": float(rd), "se": float(se),
            "ci95": (float(rd - 1.96 * se), float(rd + 1.96 * se)),
            "weighted_smd": smd, "max_abs_smd": max(abs(v) for v in smd.values())}

library(WeightIt); library(cobalt); library(survey); library(survival); library(cmprsk)

covs <- grep("^x", names(dat), value = TRUE)
f_ps <- reformulate(covs, response = "treat")

## (1) Stabilized IPTW for the ATE -------------------------------------------
w <- weightit(f_ps, data = dat, method = "glm", estimand = "ATE", stabilize = TRUE)
print(bal.tab(w, stats = "mean.diffs", thresholds = c(m = 0.1)))  # post-weighting SMDs

des  <- svydesign(ids = ~1, weights = ~w$weights, data = dat)
fit  <- svyglm(event ~ treat, design = des, family = quasibinomial())  # ITT-like risk model
print(summary(fit))                                                    # log-OR for treat

## (2) Competing-risks cumulative incidence (death as competing event) --------
## status: 0 = censored, 1 = outcome, 2 = competing death
dat$status <- with(dat, ifelse(event == 1, 1L, ifelse(death == 1, 2L, 0L)))

ci <- cuminc(ftime = dat$fu_time, fstatus = dat$status, group = dat$treat)
print(ci$Tests)   # Gray's test comparing arm-specific cumulative incidence of the outcome

fg <- crr(ftime = dat$fu_time, fstatus = dat$status,
          cov1 = model.matrix(~ treat, dat)[, -1, drop = FALSE], failcode = 1, cencode = 0)
print(summary(fg))  # Fine-Gray subdistribution hazard ratio for treat