Comparative Effectiveness Research (CER) Methods

Comparative effectiveness research (CER) methods

are the set of design and analytic tools used to recover a comparative causal estimand — the effect of treatment strategy A versus a clinically reasonable alternative B for the same decision — from real-world data when a head-to-head randomized trial cannot be run. CER is not a single estimator; it is the methods stack that sits above the specific designs and models in this catalog. This entry is the umbrella for the whole methodological enterprise (when to reach for a propensity score versus an instrument versus a g-method versus a natural experiment, and how to defend the answer for a clinical, regulatory, or HTA decision). Its sibling entry `cer-observational` is narrower: it catalogs the observational study designs (ACNU cohort, prevalent-new-user, nested case-control, self-controlled designs) and how to pick among them. Read this entry for the analytic and inferential layer; read `cer-observational` for the design-selection layer.

Core conceptual distinction

(the core estimand distinction). Three distinctions organize all of CER and must be settled in the protocol before any code runs.

1. Efficacy vs effectiveness: an RCT estimates efficacy under protocol-enforced adherence in a selected population; CER estimates effectiveness — the effect of a strategy as actually used in routine care, with imperfect adherence, broad comorbidity, and longer horizons. These are different estimands, not noisy versions of the same number.
2. Comparative vs absolute: CER compares A vs B for one decision, because a drug-vs-nothing contrast in secondary data is almost always fatally confounded by indication; the active comparator converts an unanswerable absolute question into an answerable comparative one.
3. The causal estimand itself must be named: ATE vs ATT/ATU (which the weighting or matching scheme implicitly targets — IPTW→ATE, 1:1 matching→ATT, overlap weights→ATO), and ITT-like (initiation/intention-to-treat) vs per-protocol/as-treated (which requires censoring at switch/discontinuation and inverse-probability-of-censoring weighting to handle informative censoring). Different choices imply different models and different decision-relevance; conflating them is the most common silent error in applied CER.

The method-selection logic (the heart of CER)

The driving question is what is the dominant threat to exchangeability, and is it measured?

• Confounding is measured → design out what you can (active comparator + new-user), then balance on measured covariates with a propensity score (matching, IPTW, overlap weighting, stratification) or a doubly robust estimator (PS + outcome model, e.g., AIPW/TMLE) that is consistent if either model is correct. With high-dimensional claims/EHR proxies, use a high-dimensional propensity score to recover information from variables you did not pre-specify.
• Confounding is time-varying and affected by prior treatment (a confounder that is also a mediator) → standard regression and baseline PS are biased in both directions; use g-methods (marginal structural models via IPTW, g-estimation of structural nested models, or g-computation) within a target-trial frame, typically with clone-censor-weight for sustained strategies.
• Unmeasured confounding is the dominant threat and a valid instrument exists (physician/regional preference, formulary rules, distance) → instrumental variables / 2SLS / 2SRI, accepting that the estimand is a local effect among compliers, not the population ATE, and that exclusion-restriction violations are untestable.
• A policy or natural experiment created quasi-random variation in exposure → difference-in-differences (parallel trends), regression discontinuity (a sharp eligibility threshold), or synthetic control for a single treated unit. Whatever the route, CER discipline requires pre-specification of a target trial, transparent reporting (STaRT-RWE, HARPER, the ISPOR-ISPE good-practice recommendations), fit-for-purpose data assessment, and quantitative sensitivity analysis (E-value, negative controls / empirical calibration, quantitative bias analysis) because residual confounding is never excluded by design alone.

Pros, cons, and trade-offs

• vs the head-to-head RCT it emulates: CER delivers larger, more representative populations, rare and long-term outcomes, subgroups, and answers far faster and cheaper, and it captures real-world adherence and heterogeneity that efficacy trials suppress. Cost: residual confounding (measured and unmeasured) is never excluded, the answer depends on data quality and correct estimand specification, and regulators still treat a single observational CER study as weaker than a confirmatory RCT. The RCT-DUPLICATE program showed observational CER can reproduce RCT effect estimates when design emulation is rigorous and confounders are well measured — and fails predictably when they are not. Prefer CER when an RCT is infeasible/unethical, generalizability is the question, or speed is decisive; combine with RCT evidence via transportability or network meta-analysis when possible.
• vs naive observational comparisons (prevalent-user, "ever vs never", baseline regression only): the CER stack (active comparator, new-user, PS/g-methods, target trial) removes confounding by indication, immortal time, and depletion-of-susceptibles that cripple naive analyses, and makes the estimate defensible. Cost: more complex, more assumption-heavy, and slower — and still wrong if the assumptions fail, which is exactly why sensitivity analysis is mandatory, not optional. Prefer the CER stack for essentially all non-randomized comparative questions.
• PS vs g-methods vs IV (within the stack): PS methods are simplest and best when key confounders are measured and not time-varying; g-methods are necessary (not merely better) when a covariate is both confounder and mediator of a sustained strategy; IV is the only route to unmeasured-confounding identification but pays for it with a fragile, untestable exclusion restriction and a compliers-only estimand. Reaching for IV when a good PS would do trades away precision and interpretability; reaching for a baseline PS on a time-varying problem trades away validity.

When NOT to use — and when it is actively misleading or dangerous

• No clinically interchangeable comparator exists. Forcing a comparator prescribed to systematically different patients (a second-line agent against a first-line agent) re-introduces confounding by indication and channeling — the bias you came to remove. If the honest question is drug-vs-no-treatment, the active-comparator machinery cannot answer it.
• The dominant confounder is unmeasured and no valid instrument exists. E-value or a strong negative-control association that wipes out the effect is a signal to not report a causal estimate; report the bias-analysis bound instead.
• Severe positivity violation. If one drug is reserved for the sickest patients, PS distributions separate, matching discards most of the cohort, weights explode, and the surviving estimand maps to no clinically meaningful population.
• Time-zero / immortal-time misalignment. Starting follow-up at diagnosis rather than initiation, or selecting on a post-baseline event, manufactures bias before any model is fit; no downstream adjustment repairs it.
• g-methods on thin longitudinal data. MSMs and clone-censor-weight demand correctly modeled time-varying processes and positivity at every time point; on sparse measurement they substitute model dependence for the bias they removed.

Data-source operational depth

• Claims (FFS vs MA vs commercial): the workhorse for comparative drug safety/effectiveness — large N, longitudinal fills (NDC + `fill_date` + `days_supply`), diagnoses, procedures, and costs. Strengths: near-complete utilization and pharmacy in fee-for-service. Failure modes and workarounds: Medicare Advantage encounter data are incomplete — MA-only person-time lacks reliable FFS claims, so "no prior fill" can be missingness rather than a true washout; restrict to enrollees with full Parts A/B/D (or a commercial medical+pharmacy benefit) and exclude MA-only person-time. Coding drift (ICD-9→ICD-10, fee-schedule changes) breaks longitudinal outcome algorithms — version your code lists. No labs, no severity, no reason for treatment, so channeling is invisible without linkage. Differential competing risks by exposure in elderly claims (e.g., a comparator preferentially used in frailer patients raises competing mortality) biases naive cause-specific or Kaplan-Meier outcome risk; model competing risks explicitly. 90-day mail order, sample fills, and stockpiling distort `days_supply` and on-treatment windows.
• EHR: richer covariates (labs, vitals, problem lists, notes/NLP for severity and SDoH) make effect-modification and indication confirmation possible. But initiation is the order/administration, not the dispensing (link to pharmacy to confirm the patient started), capture is visit-driven so a patient who leaves the system is differentially lost (treat loss to follow-up as potentially informative), and care outside the network is invisible.
• Registry: strongest for indication, disease severity, and adjudicated outcomes (cancer stage, device endpoints); typically weak for full pharmacy exposure and complete utilization. Link to claims for fills/costs and to a death index for censoring; excellent as the eligibility/severity backbone of a target-trial emulation.
• Linked claims–EHR–vital records: the ideal substrate (EHR severity + claims completeness + reliable mortality) but linkage introduces selection (only the linkable subset) and order/fill/service date discrepancies that must be reconciled before time-zero assignment.

Worked claims example (head-to-head, FFS logic)

Question: 2-year risk of hospitalized heart failure with a second-generation sulfonylurea vs a DPP-4 inhibitor among adults with type 2 diabetes, in a commercial + Medicare FFS database.

1. Eligibility: age ≥18, ≥2 outpatient or ≥1 inpatient diabetes diagnosis, and 365 days of continuous medical + pharmacy enrollment (full A/B/D or commercial) before the first study fill — exclude any MA-only person-time so "no prior fill" is observed, not missing.
2. Washout (new-user): no fill of any sulfonylurea or DPP-4 inhibitor in the 365-day lookback, making both arms incident users.
3. Time zero: the date of the first qualifying fill; assign the arm from the NDC dispensed that day.
4. Covariates: measured only in the 365-day window up to and including time zero (comorbidities, prior insulin, baseline HCRU, HbA1c proxies), feeding a high-dimensional propensity score.
5. Estimand and estimator: target the ATE with stabilized IPTW and add an outcome model for double robustness (AIPW), or target the ATT with 1:1 PS matching — name the choice in the SAP.
6. Follow-up and competing risks: from time zero to first validated HF hospitalization, censoring at disenrollment, end of data, and (as-treated) discontinuation (last `days_supply` end + a pre-specified grace period) or switch; treat all-cause death as a competing risk (it is differential by arm in this older cohort) and report cause-specific HR alongside the Fine-Gray subdistribution effect, since they answer different questions.
7. Diagnostics and sensitivity: standardized mean differences <0.1 after weighting/matching, an E-value for the point estimate and confidence limit, a negative-control outcome to detect residual confounding, and sensitivity to washout length and grace period.

Interpreting the output

In the new-user active-comparator design comparing Drug A and Drug B (2-year commercial claims), propensity-score balancing produces: Drug A event rate 4.2 per 100 person-years; Drug B 6.8 per 100; risk ratio ≈ 0.62; absolute risk reduction ≈ 2.6 percentage points over 2 years.

(1) Formal interpretation. The RR ≈ 0.62 is an effectiveness estimate in the real-world initiator population — patients who received prescriptions in routine care, with their actual contraindications, prior treatment failures, and adherence patterns intact. It is not an efficacy estimate from a trial's per-protocol population. The new-user restriction starts the comparison at first dispensing, eliminating prevalent-user depletion-of-susceptibles bias. The active-comparator restriction means both arms faced a similar prescribing threshold, attenuating healthy-user confounding that would inflate apparent benefit if the comparator were untreated patients. Residual confounding by indication — Drug A may be preferred in lower-risk patients — remains the primary threat to validity and should be quantified via E-value or quantitative bias analysis, alongside a negative-control outcome assessment.

(2) Practical interpretation. A 2.6 pp absolute reduction over 2 years corresponds to approximately 26 fewer events per 1,000 treated patients. This is an actionable number for coverage decisions, but carries an effectiveness caveat: the observed benefit includes the contribution of partial adherence, switching, and provider management decisions that would not replicate under controlled trial conditions. Pair the RR with a cause-specific and subdistribution hazard ratio if death is a competing event in the population.

Decision diagram

flowchart TD
  Q[Comparative question: strategy A vs B for the same decision] --> EST[Name the estimand<br/>ATE vs ATT/ATO; ITT vs per-protocol]
  EST --> TT[Pre-specify the target trial<br/>eligibility, time zero, assignment, follow-up, outcome]
  TT --> THREAT{Dominant threat to exchangeability?}
  THREAT -->|Confounding measured,<br/>not time-varying| PS[Propensity score or doubly robust<br/>matching / IPTW / overlap / AIPW / hd-PS]
  THREAT -->|Time-varying confounder<br/>that is also a mediator| G[G-methods<br/>MSM-IPTW / g-estimation / g-computation<br/>+ clone-censor-weight]
  THREAT -->|Unmeasured confounding,<br/>valid instrument exists| IV[Instrumental variable<br/>2SLS / 2SRI -> complier effect]
  THREAT -->|Policy / natural experiment| QE[Quasi-experimental<br/>DiD / RD / synthetic control]
  PS --> SENS[Quantitative sensitivity analysis<br/>E-value, negative controls, QBA]
  G --> SENS
  IV --> SENS
  QE --> SENS
  SENS --> DEC[Decision-ready comparative estimate<br/>reported per STaRT-RWE / HARPER]

flowchart LR
  subgraph Unadjusted [Naive comparison]
    T1[Treatment A vs B] -->|biased| Y1[Outcome]
    C1[Confounder by indication] --> T1
    C1 --> Y1
  end
  subgraph Adjusted [CER design + analysis]
    T2[Treatment A vs B] --> Y2[Outcome]
    C2[Measured confounders] -.balanced by PS/g-methods.-> T2
    C2 --> Y2
    U[Unmeasured confounding] -->|probed by E-value / negative controls| Y2
    U --> T2
  end

Worked example

Scenario

A payer wants to know whether Drug A (a newer diabetes pill) leads to fewer hospitalizations for heart failure than Drug B (an older diabetes pill) among adults who are just starting one of these two drugs. No head-to-head trial has been run. Researchers build an active-comparator, new-user study from two years of commercial insurance claims.

Dataset

Five design decisions a CER analyst documents before running any code — one row per choice, contrasting effectiveness in the real world vs what an efficacy trial would do.

Steps

Result

After propensity-score balancing, the Drug A group and Drug B group look similar on all measured baseline characteristics. The 2-year heart failure hospitalization rate is 4.2 per 100 patients in the Drug A arm versus 6.8 per 100 in the Drug B arm — a 2.6 percentage-point absolute reduction (relative risk 0.62). Because this is measured in routine care with real-world adherence and a broad patient population, it is an effectiveness estimate, not the efficacy estimate a placebo-controlled trial would produce.

Trade-offs

Runnable example

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression

def aipw_ate(df: pd.DataFrame, covs: list[str], horizon_days: int = 730) -> dict:
    """Doubly robust ATE (risk difference) for a binary outcome by `horizon_days`.

    AIPW is consistent if EITHER the propensity model OR the outcome model is correct.
    Death-without-event is treated as a censoring competing event: subjects who die
    before the horizon without the outcome and with fu_time < horizon are censored, so
    the contrast is the cause-specific (treatment-as-cause) risk difference. For a
    policy-relevant cumulative-incidence (subdistribution) contrast, switch to a
    Fine-Gray / Aalen-Johansen estimator (see the R and SAS blocks).
    """
    d = df.copy()
    # Outcome observed by the horizon (event before horizon AND followed long enough).
    d["y"] = ((d["event"] == 1) & (d["fu_time"] <= horizon_days)).astype(int)

    X = d[covs].to_numpy()
    a = d["treat"].to_numpy()
    y = d["y"].to_numpy()

    # --- Propensity model -> stabilized IPTW ---
    ps = LogisticRegression(max_iter=1000).fit(X, a).predict_proba(X)[:, 1]
    ps = np.clip(ps, 0.01, 0.99)                 # bound to tame extreme weights
    p_treat = a.mean()
    sw = np.where(a == 1, p_treat / ps, (1 - p_treat) / (1 - ps))  # stabilized weights

    # --- Outcome model (fit within each arm to allow effect modification) ---
    mu1 = LogisticRegression(max_iter=1000).fit(X[a == 1], y[a == 1]).predict_proba(X)[:, 1]
    mu0 = LogisticRegression(max_iter=1000).fit(X[a == 0], y[a == 0]).predict_proba(X)[:, 1]

    # --- AIPW influence-function estimates of E[Y^1] and E[Y^0] ---
    psi1 = mu1 + (a / ps) * (y - mu1)
    psi0 = mu0 + ((1 - a) / (1 - ps)) * (y - mu0)
    rd = psi1.mean() - psi0.mean()
    se = np.sqrt(np.var(psi1 - psi0, ddof=1) / len(d))  # influence-function SE

    # --- Weighted covariate balance (standardized mean differences) ---
    def wsmd(col):
        t, c = d[a == 1], d[a == 0]
        wt, wc = sw[a == 1], sw[a == 0]
        mt = np.average(t[col], weights=wt); mc = np.average(c[col], weights=wc)
        vt = np.average((t[col] - mt) ** 2, weights=wt)
        vc = np.average((c[col] - mc) ** 2, weights=wc)
        return (mt - mc) / np.sqrt((vt + vc) / 2 + 1e-12)

    smd = {c: round(float(wsmd(c)), 3) for c in covs}
    return {"risk_diff": float(rd), "se": float(se),
            "ci95": (float(rd - 1.96 * se), float(rd + 1.96 * se)),
            "weighted_smd": smd, "max_abs_smd": max(abs(v) for v in smd.values())}

library(WeightIt); library(cobalt); library(survey); library(survival); library(cmprsk)

covs <- grep("^x", names(dat), value = TRUE)
f_ps <- reformulate(covs, response = "treat")

## (1) Stabilized IPTW for the ATE -------------------------------------------
w <- weightit(f_ps, data = dat, method = "glm", estimand = "ATE", stabilize = TRUE)
print(bal.tab(w, stats = "mean.diffs", thresholds = c(m = 0.1)))  # post-weighting SMDs

des  <- svydesign(ids = ~1, weights = ~w$weights, data = dat)
fit  <- svyglm(event ~ treat, design = des, family = quasibinomial())  # ITT-like risk model
print(summary(fit))                                                    # log-OR for treat

## (2) Competing-risks cumulative incidence (death as competing event) --------
## status: 0 = censored, 1 = outcome, 2 = competing death
dat$status <- with(dat, ifelse(event == 1, 1L, ifelse(death == 1, 2L, 0L)))

ci <- cuminc(ftime = dat$fu_time, fstatus = dat$status, group = dat$treat)
print(ci$Tests)   # Gray's test comparing arm-specific cumulative incidence of the outcome

fg <- crr(ftime = dat$fu_time, fstatus = dat$status,
          cov1 = model.matrix(~ treat, dat)[, -1, drop = FALSE], failcode = 1, cencode = 0)
print(summary(fg))  # Fine-Gray subdistribution hazard ratio for treat

/* Composite competing-risks status: 0=censored, 1=outcome, 2=competing death. */
data analytic;
  set work.analytic;
  if event = 1 then crstatus = 1;
  else if death = 1 then crstatus = 2;
  else crstatus = 0;
run;

/* (a) Inverse-probability-of-treatment weighting for the ATE, with balance output. */
proc psmatch data=analytic region=allobs;
  class treat;
  psmodel treat(treated='1') = x1 x2 x3 x4;        /* baseline-window covariates only */
  assess lps var=(x1 x2 x3 x4) / weight=atewgt;     /* standardized differences pre/post */
  output out(obs=all)=wtd atewgt=ipw;
run;

/* IPTW outcome model on the binary endpoint (ITT-like risk contrast). */
proc genmod data=wtd descending;
  class treat / param=ref ref='0';
  weight ipw;
  model event = treat / dist=binomial link=logit;
  estimate 'Study vs Comparator (log OR)' treat 1 -1;
run;

/* (b) Doubly robust (AIPW) ATE on the risk-difference scale. */
proc causaltrt data=analytic method=aipw;
  class treat;
  psmodel treat(event='1') = x1 x2 x3 x4;
  model event = x1 x2 x3 x4;                         /* outcome model -> double robustness */
run;

/* (c) Competing-risks: Fine-Gray subdistribution HR + cause-specific cumulative incidence. */
proc phreg data=analytic;
  class treat (ref='0');
  model fu_time*crstatus(0) = treat / eventcode=1 ties=efron;   /* Fine-Gray for the outcome */
  hazardratio treat / diff=ref;
run;

proc lifetest data=analytic plots=cif(test);
  time fu_time*crstatus(0) / eventcode=1;            /* cumulative incidence accounting for death */
  strata treat;
run;

Citations

1. [1]Johnson ML, Crown W, Martin BC, Dormuth CR, Siebert U. Good research practices for comparative effectiveness research: analytic methods to improve causal inference from nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report-Part III. Value in Health. 2009;12(8):1062-1073.
2. [2]Berger ML, Sox H, Willke RJ, et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making. Pharmacoepidemiol Drug Saf. 2017;26(9):1033-1039.
3. [3]Hernán MA, Sauer BC, Hernández-Díaz S, Platt R, Shrier I. Specifying a target trial prevents immortal time bias and other self-inflicted injuries in observational analyses. J Clin Epidemiol. 2016;79:70-75.
4. [4]Schneeweiss S, Rassen JA, Glynn RJ, Avorn J, Mogun H, Brookhart MA. High-dimensional propensity score adjustment in studies of treatment effects using health care claims data. Epidemiology. 2009;20(4):512-522.

1. [5]Franklin JM, Patorno E, Desai RJ, et al. Emulating randomized clinical trials with nonrandomized real-world evidence studies: first results from the RCT DUPLICATE initiative. Circulation. 2021;143(10):1002-1013.

1. [6]Daigl M, Abogunrin S. Advancing the role of real-world evidence in comparative effectiveness research. J Comp Eff Res. 2024;13(7):e240101.