Observational Comparative Effectiveness Research

Observational comparative effectiveness research (CER)

asks which of two (or more) clinically reasonable treatment strategies works better, or is safer, in routine care — using data generated by clinical practice rather than by randomization. It is the umbrella under which specific non-randomized designs live: the active-comparator new-user (ACNU) cohort, the prevalent-new-user cohort, the nested case-control study, self-controlled designs, and explicit target-trial emulation. The defining feature is not a single procedure but a comparative estimand recovered without randomization, which means every design choice exists to reconstruct, from observational data, the three things randomization gives you for free: a well-defined eligible population, a sharp time zero, and exchangeable treatment groups.

Core conceptual distinction — effectiveness vs efficacy, and comparative vs absolute

A randomized trial estimates efficacy: the effect of a treatment under protocol-enforced adherence in a selected, consenting population. Observational CER estimates effectiveness: the effect of a treatment strategy as actually used — imperfect adherence, broad comorbidity, real prescribing patterns, longer horizons. These are different estimands, not noisy versions of the same number, and the gap (the "efficacy-effectiveness gap") is the substantive reason CER exists. The second distinction is comparative vs absolute: observational CER is built to compare strategy A vs strategy B for the same decision, because a drug-vs-nothing contrast in secondary data is almost always fatally confounded by indication. Choosing a clinically interchangeable active comparator is what converts an unanswerable absolute question into an answerable comparative one.

The design spectrum within observational CER (how to pick)

• Active-comparator, new-user (ACNU) cohort — the default for chronic-therapy head-to-head questions. Both arms initiate a drug for the same indication after a drug-free washout; time zero is initiation. Use when a clinically interchangeable comparator and an initiation event both exist.
• Prevalent-new-user cohort (Suissa) — when initiation of the study drug is too rare to support an incident-user cohort, this matches new initiators to time-matched prevalent users of the comparator. Use when ACNU starves for power.
• Nested case-control — when outcomes are rare and full-cohort covariate construction (e.g., expensive chart abstraction or biomarker assays) is infeasible; sample controls by risk-set sampling to recover the rate ratio efficiently.
• Self-controlled designs (SCCS, case-crossover) — when time-invariant confounding dominates (genetics, chronic severity) and the exposure is transient with an acute outcome. Each person is their own control, eliminating between-person confounding entirely — but they cannot handle time-varying confounders or estimate a between-drug contrast.
• Target-trial emulation — the organizing framework, not a separate data structure: write the protocol of the trial you wish you could run (eligibility, treatment strategies, assignment, time zero, outcome, estimand, analysis), then map each element to the data. Use it always as the discipline; reach for clone-censor-weight when the strategies are sustained or dynamic and grace periods create eligibility-time ambiguity. The single most common, most expensive error in observational CER is not picking the wrong model — it is misaligning time zero (the eligibility, exposure, and follow-up start), which silently manufactures immortal-time and selection bias before any estimator runs.

Pros, cons, and trade-offs

• vs the randomized controlled trial: Observational CER answers questions an RCT cannot or will not — head-to-head comparisons no sponsor will fund, broad/elderly/multimorbid populations excluded from trials, long-horizon and rare safety outcomes, and post-launch effectiveness. Cost: it carries unmeasured confounding that randomization removes by design; positivity and exchangeability are assumptions you must defend, not guarantees. Prefer the RCT when it is feasible and ethical and the question is efficacy; prefer observational CER when an RCT is infeasible, too slow, too narrow, or already answered efficacy and the live question is real-world effectiveness.
• vs the pragmatic / registry-based RCT: A pragmatic trial keeps randomization (so confounding is handled) while relaxing protocol toward routine care. Cost: it is slow, costly, and still selects consenting sites/patients. Prefer a pragmatic trial when randomization is achievable and equipoise exists; prefer observational CER when randomization is impossible, unethical, or far too slow for the decision timeline.
• vs single-arm study with external/historical control: Observational CER compares two concurrent arms from the same source, sharing secular trends and measurement. Cost: it needs a real comparator population, which may not exist for an ultra-rare disease or a first-in-class agent. Prefer concurrent comparative CER whenever a contemporaneous comparator exists; fall back to an external control only when it genuinely does not.

When NOT to use — and when it is actively misleading or dangerous

• No clinically interchangeable comparator exists. Forcing a comparator prescribed to systematically different patients (second-line vs first-line; sicker vs healthier) re-introduces confounding by indication and channeling — the exact bias CER is meant to remove. The result can be precisely wrong and worse than no evidence.
• The real question is drug vs no treatment (uptake, deprescribing, the effect of being treated at all). Active-comparator CER structurally cannot answer it, and a non-user comparator in claims is almost always confounded by indication and healthy-user bias.
• Exposure or outcome is poorly captured in the data. If the outcome algorithm has low PPV/sensitivity, or exposure is invisible (samples, inpatient-administered drugs absent from pharmacy claims, OTC use), differential misclassification can create or hide an effect. Validate the algorithm before trusting the comparison, not after.
• Severe positivity violation / non-overlap. If one strategy is reserved for sicker or renally-impaired patients, propensity distributions separate; matching discards most of the cohort and the surviving estimand no longer maps to any decision-relevant population.
• Strong time-varying confounding affected by prior treatment (e.g., labs that drive both subsequent dosing and the outcome). Standard PS/regression adjustment is biased here; this requires g-methods (MSM/IPTW, g-estimation), and if the team cannot implement them, the naive comparative estimate is misleading.

Data-source operational depth

• Claims (FFS or commercial): Exposure is the pharmacy claim (NDC + `fill_date` + `days_supply`); diagnosis/procedure codes define indication, covariates, and many outcomes. Strengths: near-complete capture of dispensed drugs and billed encounters within a covered, continuously enrolled population. Failure modes: (1) Medicare Advantage / capitated person-time lacks fee-for-service claims — "no prior diagnosis" or "no prior fill" can be missingness, not truth; restrict to enrollees with the relevant benefit (A/B/D or commercial medical+pharmacy) and exclude MA-only person-time. (2) Differential competing risks by exposure in elderly claims — if one drug is preferentially given to frailer patients, death competes with the outcome differently across arms; use cause-specific or Fine-Gray models rather than naive Kaplan-Meier. (3) Inpatient-administered drugs are invisible to pharmacy claims; sample fills, 90-day mail order, and stockpiling distort `days_supply`-based episodes. Workaround: continuous-enrollment requirements across the full lookback, explicit episode/grace-period rules, and validated outcome algorithms.
• EHR: Initiation is the order or administration, not the dispensing — link to pharmacy fills to confirm the patient actually started. Problem lists, labs, vitals, and notes sharpen indication and baseline severity (a real advantage over claims), but visit-driven capture means a patient who seeks care outside the system is differentially lost; define observation windows explicitly and treat loss to follow-up as potentially informative. Care fragmentation is the signature EHR failure mode in CER.
• Registry: Strongest for indication, disease severity/stage, and adjudicated outcomes; typically weak for complete pharmacy exposure and for non-registry comorbidity. Link to claims for the full fill history and to a death index to firm up censoring. The signature failure mode is enrollment selection and incomplete capture of out-of-registry events.
• Linked claims-EHR-vital records: The ideal substrate — EHR severity + claims completeness + reliable mortality — but linkage introduces selection (only the linkable subset, which differs from the source population) and order/fill/service date discrepancies that must be reconciled before time-zero assignment. The failure mode is treating the linked subset as representative.

Worked claims example (a CER design choice, not just a cohort build). Question: among adults with type 2 diabetes, does an SGLT2 inhibitor reduce heart-failure hospitalization compared with a DPP-4 inhibitor? In a commercial + Medicare FFS claims database, walk the decision:

1. Is there an interchangeable active comparator? Yes — both are added at a similar point in the treatment pathway, so an ACNU cohort is defensible (a non-user comparator would be confounded by indication and is rejected).
2. Is initiation common enough for an incident-user cohort? Yes — so ACNU, not prevalent-new-user.
3. Eligibility: age ≥18, ≥2 diabetes diagnoses, and 365 days of continuous A/B/D (or commercial medical+pharmacy) enrollment before the first study fill, excluding MA-only person-time so absence of prior fills is observed rather than missing.
4. Washout / time zero: no fill of either class in the 365-day lookback makes both arms incident users; time zero is the first qualifying fill, and the arm is read from the NDC dispensed that day.
5. Outcome: a validated HF-hospitalization algorithm (inpatient claim with HF in the primary position) — its PPV is checked, not assumed.
6. Confounding: baseline covariates measured only in [time zero − 365, time zero] feed a high-dimensional propensity score; balance is confirmed with standardized differences <0.1.
7. Competing risk: because the cohort skews elderly, death is modeled as a competing event (cause-specific hazard for etiology, Fine-Gray for absolute risk).
8. Sensitivity: negative-control outcomes and an E-value quantify residual confounding; washout and grace-period lengths are varied. The deliverable is a comparative hazard/risk contrast with a defensible target-trial mapping — the design decisions, not the regression, are where the evidence is won or lost.

Decision diagram

flowchart TD
  Q[Comparative decision:<br/>strategy A vs strategy B] --> RCT{RCT feasible,<br/>ethical, timely?}
  RCT -- Yes --> DoRCT[Run RCT / pragmatic trial<br/>randomization handles confounding]
  RCT -- No --> COMP{Clinically interchangeable<br/>active comparator exists?}
  COMP -- No, real question is<br/>drug vs no treatment --> STOP[Observational CER not valid here<br/>confounding by indication]
  COMP -- No comparator pop.<br/>ultra-rare / first-in-class --> EXT[Single-arm + external control]
  COMP -- Yes --> INIT{Initiation common enough<br/>for incident users?}
  INIT -- No --> PNU[Prevalent-new-user cohort]
  INIT -- Yes --> RARE{Outcome rare or<br/>covariates costly?}
  RARE -- Yes --> NCC[Nested case-control]
  RARE -- No --> TV{Time-varying confounding<br/>affected by prior treatment?}
  TV -- Yes --> GM[Target-trial emulation +<br/>g-methods / clone-censor-weight]
  TV -- No --> ACNU[Active-comparator new-user cohort<br/>+ propensity score]

flowchart LR
  Indication[Indication / disease severity] --> Tx[Treatment choice:<br/>strategy A vs B]
  Indication --> Y[Outcome]
  Tx --> Y
  Confounders[Measured + unmeasured<br/>baseline confounders] --> Tx
  Confounders --> Y
  Indication -. confounding by indication .-> Tx
  classDef bias fill:#fde,stroke:#b36;
  class Indication bias

Worked example

Scenario

Among adults with type 2 diabetes, we want to know whether starting drug A (an SGLT2 inhibitor) is associated with fewer heart-failure hospitalizations than starting drug B (a DPP-4 inhibitor) over one year of follow-up. We pull two groups of new starters from a claims database, count how many in each group were hospitalized for heart failure, and compare the two event rates head to head.

Dataset

One summary row per treatment arm, built from a claims cohort: how many patients started each drug and how many had a heart-failure hospitalization within the year.

Steps

Result

Drug A had a heart-failure-hospitalization risk of 30/500 = 0.060 versus 40/400 = 0.100 for drug B. Risk difference = 0.060 − 0.100 = −0.040 (4.0 fewer events per 100 patients); risk ratio = 0.060 / 0.100 = 0.60 (a 40% relative reduction) — an estimate that is only trustworthy if the two arms were comparable at baseline.

Trade-offs

Runnable example

import pandas as pd

WASHOUT_DAYS = 365  # drug-free + continuous, FFS-observable lookback that defines an incident user

def build_cer_cohort(rx: pd.DataFrame, enroll: pd.DataFrame) -> pd.DataFrame:
    rx = rx.sort_values(["person_id", "fill_date"])
    strat = rx[rx["drug_class"].isin(["STUDY", "COMPARATOR"])]

    # Candidate time zero = first fill of EITHER strategy; arm = the class dispensed that day.
    idx = (strat.groupby("person_id").first().reset_index()
                .rename(columns={"fill_date": "index_date", "drug_class": "arm"}))

    # New-user restriction: no fill of either strategy in the washout window before index.
    prior = strat.merge(idx[["person_id", "index_date"]], on="person_id")
    had_prior = prior[(prior["fill_date"] < prior["index_date"]) &
                      (prior["fill_date"] >= prior["index_date"] - pd.Timedelta(days=WASHOUT_DAYS))]
    idx = idx[~idx["person_id"].isin(had_prior["person_id"])].copy()

    # Continuous, FFS-observable enrollment spanning the full washout through index
    # (so "no prior fill/dx" is truly observed, not MA/capitated missingness).
    e = enroll.merge(idx[["person_id", "index_date"]], on="person_id")
    e["covers"] = ((e["enroll_start"] <= e["index_date"] - pd.Timedelta(days=WASHOUT_DAYS)) &
                   (e["enroll_end"]   >= e["index_date"]) & e["ffs_observable"])
    eligible = e.loc[e["covers"], "person_id"].unique()

    cohort = idx[idx["person_id"].isin(eligible)].copy()
    cohort["baseline_start"] = cohort["index_date"] - pd.Timedelta(days=WASHOUT_DAYS)  # covariate window
    return cohort[["person_id", "arm", "index_date", "baseline_start"]]

library(data.table)
WASHOUT_DAYS <- 365L

build_cer_cohort <- function(rx, enroll) {
  setDT(rx); setDT(enroll)
  setorder(rx, person_id, fill_date)

  strat <- rx[drug_class %chin% c("STUDY", "COMPARATOR")]
  idx <- strat[, .(index_date = fill_date[1L], arm = drug_class[1L]), by = person_id]

  # New-user: drop anyone with a study/comparator fill in the washout window before index.
  strat <- merge(strat, idx[, .(person_id, index_date)], by = "person_id")
  prior_ids <- unique(strat[fill_date < index_date &
                            fill_date >= index_date - WASHOUT_DAYS, person_id])
  idx <- idx[!person_id %chin% prior_ids]

  # Continuous, FFS-observable enrollment across the full washout through index.
  e <- merge(enroll, idx[, .(person_id, index_date)], by = "person_id")
  ok <- e[enroll_start <= index_date - WASHOUT_DAYS &
          enroll_end   >= index_date & ffs_observable, unique(person_id)]

  cohort <- idx[person_id %chin% ok]
  cohort[, baseline_start := index_date - WASHOUT_DAYS]
  cohort[, .(person_id, arm, index_date, baseline_start)]
}

%let washout = 365;

/* First fill of either strategy = candidate time zero, with the arm dispensed that day. */
proc sort data=work.rx(where=(drug_class in ('STUDY','COMPARATOR'))) out=rx_q;
  by person_id fill_date;
run;

data idx;
  set rx_q;
  by person_id;
  if first.person_id;
  index_date = fill_date;
  format index_date date9.;
  length arm $12;
  arm = drug_class;
  keep person_id index_date arm;
run;

/* New-user restriction: exclude any prior study/comparator fill inside the washout window. */
proc sql;
  create table newuser as
  select i.*
  from idx i
  where not exists (
    select 1 from work.rx p
    where p.person_id = i.person_id
      and p.drug_class in ('STUDY','COMPARATOR')
      and p.fill_date <  i.index_date
      and p.fill_date >= i.index_date - &washout
  );
quit;

/* Continuous, FFS-observable enrollment across the full washout through index (no MA/capitated gaps). */
proc sql;
  create table cohort as
  select n.person_id, n.arm, n.index_date,
         n.index_date - &washout as baseline_start format=date9.
  from newuser n
  where exists (
    select 1 from work.enroll e
    where e.person_id = n.person_id
      and e.ffs_observable = 1
      and e.enroll_start <= n.index_date - &washout
      and e.enroll_end   >= n.index_date
  );
quit;

Citations

1. [1]Schneeweiss S, Avorn J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. Journal of Clinical Epidemiology. 2005;58(4):323-337.
2. [2]Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. American Journal of Epidemiology. 2016;183(8):758-764.
3. [3]Berger ML, Sox H, Willke RJ, et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the joint ISPOR-ISPE Special Task Force on Real-World Evidence in Health Care Decision Making. Value in Health. 2017;20(8):1003-1008.
4. [4]Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidence — what is it and what can it tell us? New England Journal of Medicine. 2016;375(23):2293-2297.

1. [5]Schneeweiss S, Rassen JA, Glynn RJ, et al. High-dimensional propensity score adjustment in studies of treatment effects using health care claims data. Epidemiology. 2009;20(4):512-522.

1. [6]Dreyer NA, Bryant A, Velentgas P. The GRACE checklist: a validated assessment tool for high quality observational studies of comparative effectiveness. Journal of Managed Care & Specialty Pharmacy. 2016;22(10):1107-1113.