Benchmarking Target-Trial Emulations Against Randomized Trials

RCT benchmarking

compares an observational emulation with a randomized trial that answers a sufficiently similar question. It is a design-and-measurement diagnostic, not a test that turns the observational estimate into causal truth. The comparison has meaning only after both studies are mapped to the same target question: population, treatment strategy, comparator, assignment/time zero, outcome definition, follow-up, causal contrast, and effect scale. A hazard ratio from an as-treated database cohort is not directly benchmarked to an intention-to-treat trial hazard ratio merely because the drug names match.

Prespecification comes first

Before estimating the emulation effect, freeze the trial protocol abstraction, data mappings, primary estimand, confounding strategy, expected direction and practical agreement metric. Define the comparison metric in the estimand's natural scale (for example, absolute risk difference at 365 days, log risk ratio, or survival difference), with an uncertainty interval. Do not use overlapping confidence intervals or two nonsignificant p values as the sole definition of agreement: both can reflect imprecision rather than substantive concordance. A practical tolerance should be justified by clinical or decision relevance and reported alongside the signed difference and both estimates.

Interpret discordance diagnostically

First audit trial/emulation mismatches in eligibility, baseline risk, initiation versus assignment, adherence handling, outcome ascertainment, competing events, loss to follow-up, and effect-measure scale. Then examine measured confounding and positivity in the database. Disagreement may reflect residual confounding, data error, selection, an inequivalent intervention, real population effect modification, or random error; it does not identify which mechanism caused it. Agreement is reassuring for the specific design, setting, endpoint, and contrast, but cannot prove absence of unmeasured confounding in a new study.

When not to use

Do not call any observational study a trial replication when key trial eligibility, exposure, outcome, or confounders are absent from the data. Do not force agreement by choosing analytic specifications after seeing the trial result. A benchmark trial with a different intervention version, calendar era, clinical setting, or target population may be useful only after an explicit transportability assessment. No benchmark RCT does not mean an emulation is invalid; it means this external diagnostic is unavailable.

Pros, cons, and trade-offs

Benchmarking provides an external randomized reference for a carefully aligned question. It costs detailed protocol mapping and may be infeasible when important eligibility or outcome measures are absent. Successful agreement supports that setting but does not establish validity for another population, endpoint, or intervention.

Data-source operational depth

• Claims: paid dispensings approximate initiation but not randomized assignment or ingestion. Verify whether trial inclusion rules, mortality, and outcomes are observable through the same follow-up horizon.
• EHR: map protocol outcomes and visit windows without conditioning on post-baseline care intensity; external care and missing outcome assessments can create a benchmark mismatch.
• Registry: adjudicated trial-like outcomes help, but registry enrollment and site mix can select a different population from the RCT.
• Linked data: linkage can improve covariate and outcome fidelity but defines a selected analytic subset that must be compared with both the source database and the trial population.

Decision diagram

%%{init: {'themeVariables': {'fontSize': '16px'}, 'flowchart': {'nodeSpacing': 18, 'rankSpacing': 24}}}%%
flowchart TD
  A["RCT protocol and data<br/>Define the same target question"] --> B["Align population, treatment,<br/>time zero, outcome, and estimand"]
  B --> C["Prespecify design<br/>and agreement tolerance"]
  C --> D["Run observational emulation<br/>without result-driven tuning"]
  D --> E["Compare on the same scale<br/>with uncertainty"]
  E --> F["Interpret separately<br/>Agreement: context-specific<br/>Discordance: investigate<br/>Imprecise: unresolved"]

Worked example

Scenario

A synthetic claims emulation is benchmarked to a completed randomized new-user trial of Drug A versus Drug B for one-year hospitalization risk. Both analyses target a 365-day initiation-strategy risk difference in adults meeting the trial's baseline eligibility criteria.

Dataset

Prespecified comparison summary; numbers are synthetic and do not represent a clinical result.

Steps

• Abstract the RCT protocol and create an alignment table for eligibility, treatment definitions, time zero, follow-up, outcome, censoring, and estimand before looking at estimates.

• Freeze the database phenotype, confounder adjustment plan, and a practical agreement threshold of plus or minus 2 percentage points (0.02 absolute risk difference).

• Estimate the emulation's standardized 365-day risks and compare the signed risk-difference gap with the RCT estimate and its uncertainty.

• Audit Any Mismatch Before Changing The Analysis

  check population composition, treatment ascertainment, outcome capture, censoring, competing events, and support.

• Report the comparison as evidence about this emulation's fidelity, not as validation of causal claims in a different study.

Result

The point estimates agree exactly, but with independent-study standard errors of 0.01 and 0.015 the 95% interval for their gap is about -3.5 to 3.5 percentage points. The point is within the 2-point tolerance but the interval is not wholly within it; this is not conclusive practical agreement.

Trade-offs

Code example

import pandas as pd

def benchmark_summary(results: pd.DataFrame, tolerance: float = 0.02) -> dict:
    # Required: study ('RCT'/'emulation'), estimate, se; same estimand and scale.
    rct = results.loc[results.study.eq("RCT")].iloc[0]
    emu = results.loc[results.study.eq("emulation")].iloc[0]
    gap = float(emu.estimate - rct.estimate)
    se_gap = float((emu.se ** 2 + rct.se ** 2) ** 0.5)  # assumes independent estimates
    ci = (gap - 1.96 * se_gap, gap + 1.96 * se_gap)
    return {"signed_gap": gap, "gap_95_ci": ci,
            "point_within_tolerance": abs(gap) <= tolerance,
            "ci_wholly_within_tolerance": ci[0] >= -tolerance and ci[1] <= tolerance,
            "note": "Diagnostic only; independence and protocol alignment require justification."}

results = pd.DataFrame({"study": ["RCT", "emulation"],
                        "estimate": [-0.03, -0.03], "se": [0.01, 0.015]})
print(benchmark_summary(results))  # reports the prespecified diagnostic summary

benchmark_summary <- function(results, tolerance = 0.02) {
  rct <- subset(results, study == "RCT")[1, ]
  emu <- subset(results, study == "emulation")[1, ]
  gap <- emu$estimate - rct$estimate
  se_gap <- sqrt(emu$se^2 + rct$se^2) # assumes independent estimates
  ci <- gap + c(-1, 1) * 1.96 * se_gap
  list(signed_gap = gap, gap_95_ci = ci,
       point_within_tolerance = abs(gap) <= tolerance,
       ci_wholly_within_tolerance = ci[1] >= -tolerance && ci[2] <= tolerance)
}
results <- data.frame(study=c("RCT", "emulation"), estimate=c(-.03, -.03),
                      se=c(.01, .015))
print(benchmark_summary(results)) # reports the prespecified diagnostic summary

%let tolerance=0.02;
data work.results;
  length study $ 9;
  input study $ estimate se;
  datalines;
RCT -0.03 0.01
emulation -0.03 0.015
;
run;
proc sql;
  create table benchmark as
  select e.estimate-r.estimate as signed_gap,
         sqrt(e.se**2+r.se**2) as se_gap,
         calculated signed_gap-1.96*calculated se_gap as gap_lcl,
         calculated signed_gap+1.96*calculated se_gap as gap_ucl,
         (abs(calculated signed_gap) <= &tolerance) as point_within_tolerance,
         (calculated gap_lcl >= -&tolerance and calculated gap_ucl <= &tolerance) as ci_wholly_within_tolerance
  from work.results as r, work.results as e
  where r.study='RCT' and e.study='emulation';
quit;

Citations

1. [1]Franklin JM, Pawar A, Martin D, et al. Nonrandomized real-world evidence to support regulatory decision making: Process for a randomized trial replication project. Clinical Pharmacology & Therapeutics. 2020;107(4):817-826.
2. [2]Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. American Journal of Epidemiology. 2016;183(8):758-764.

1. [3]Wang SV, Schneeweiss S, RCT-DUPLICATE Initiative. Emulation of randomized clinical trials with nonrandomized database analyses: Results of 32 clinical trials. JAMA. 2023;329(16):1376-1385.