Regulatory and HTA Readiness for RWE

In plain language

Regulatory readiness is the practice of building a real-world evidence study so that every decision — what data you used, what question you were answering, how the analysis was planned — is written down, locked, and traceable before you touch the data. Think of it as the paperwork and process checks a study must pass before the FDA, EMA, or a health-technology assessment body will trust its results. Without it, a technically sound analysis can still be rejected because no one can verify that the answer wasn't shaped by peeking at the results first. A ready study is one where an independent analyst could take your documentation package and re-derive your headline number from scratch.

Regulatory and HTA readiness

is the practice of engineering a real-world evidence (RWE) study so that the finished package — protocol, statistical analysis plan (SAP), data-provenance documentation, code, and report — can withstand structured review by a regulator (FDA, EMA) or a health technology assessment body (NICE, ICER, IQWiG, CADTH/CDA) before the study is run, not after the estimate is in hand. It is not a single statistical method; it is the connective tissue that forces fitness-for-purpose, estimand traceability, time-zero alignment, comparator defensibility, and pre-specified sensitivity analyses into a single auditable chain. Readiness operationalizes the structured-template movement — STaRT-RWE for planning and reporting, HARPER for hypothesis-evaluating treatment-effect studies, ICH E9(R1) for estimands — and the target-trial framework that disciplines the design.

Core conceptual distinction

Readiness is a process and documentation layer, not an estimation layer. The estimation methods (active comparator new-user, propensity scores, Cox/Fine-Gray, g-methods) live in their own catalog entries; readiness asks a different question of each: is the choice pre-specified, justified against the policy question, traceable from estimand to code, and accompanied by a falsification/sensitivity strategy that a skeptical reviewer would accept? The decisive distinction a reviewer enforces is between a study that reports an estimate and a study that can defend the estimand it claims to have measured — the population, the treatment strategies being contrasted, the outcome, the timing of intercurrent events, and the summary measure, in ICH E9(R1) terms. A study can be statistically flawless and still fail readiness if the data cannot observe the estimand (e.g., a per-protocol estimand built on claims that cannot see in-hospital administration), or if the analysis was specified after looking at results. Readiness is therefore upstream of, and orthogonal to, the precision of any one estimator.

Pros, cons, and trade-offs

- vs running the analysis first and writing the protocol around it ("HARKing"/post hoc rationalization): Readiness front-loads pre-specification and a locked SAP, which is the single strongest defense against the "you fished for this result" critique that sinks RWE submissions. Cost: it is slow, demands cross-functional sign-off, and constrains exploratory creativity. Prefer readiness for any confirmatory or label-relevant submission; relax it only for genuinely hypothesis-generating internal work that will be re-run confirmatorily. - vs a generic study protocol with no structured template: Adopting STaRT-RWE/HARPER plus an estimand-to-code traceability matrix makes the package machine-checkable and reviewer-navigable, and exposes gaps (an unjustified comparator, an undocumented washout) early. Cost: template overhead and the discipline of keeping the matrix current as the protocol evolves. Prefer the structured template whenever a regulator or HTA body is the audience. - vs treating fit-for-purpose data assessment as a checkbox: Real readiness audits provenance, relevance, and reliability against the specific estimand (can these data observe time zero, the comparator, the outcome with adequate PPV, and complete person-time?). Cost: it can disqualify a convenient data source late. Prefer the deep assessment — a late "the data can't see the outcome" is far cheaper before lock than after submission. - vs deferring the design to a target-trial emulation alone: Target-trial emulation supplies the design scaffold (eligibility, treatment strategies, assignment, time zero, causal contrast) that readiness then documents and stress-tests. They are complements, not substitutes; readiness is the wrapper that turns a well-emulated trial into a submittable dossier with negative controls, quantitative bias analysis, and a transparency/reproducibility package.

When to use

Any study intended to support a regulatory action (label expansion, post-marketing requirement/PASS, safety signal evaluation) or an HTA submission (relative effectiveness, budget impact, survival extrapolation inputs); any externally controlled trial or single-arm-plus-external-control submission; any study where a payer or regulator will independently re-derive the result from your code and data dictionary. Engage readiness at protocol conception, not at write-up: the cheapest defects to fix are the ones caught before data lock.

When NOT to use — and when it is actively misleading or dangerous

Do not impose the full regulatory-readiness apparatus on genuinely exploratory, internal-only analyses whose purpose is to decide whether a confirmatory study is worth running — the overhead can kill useful hypothesis generation, and a locked SAP on an exploratory question is theater. It becomes actively misleading when used as a veneer: a beautifully templated protocol wrapped around a data source that cannot observe the estimand gives reviewers false confidence and is worse than an honestly limited study, because the polish hides the fatal flaw. It is dangerous when "readiness" is invoked to justify a feasibility-driven estimand swap — silently redefining the question to whatever the data can answer (e.g., switching a drug-vs-no-treatment policy question to a drug-vs-active-comparator question because the comparator was convenient) while presenting it as the original question. Readiness must protect the question, not bend it to the data.

Data-source operational depth

- Administrative claims (FFS vs Medicare Advantage vs commercial): The dominant readiness failure is unobservable person-time. Medicare Advantage enrollees generate no fee-for-service claims, so apparent "no prior fill" during a washout can be encounter-data gaps rather than a true drug-free period; restrict to enrollees with complete Parts A/B/D (or a commercial medical+pharmacy benefit) and exclude MA-only person-time, and document this in the fit-for-purpose section. Claims can establish exposure (NDC + `fill_date` + `days_supply`) and outcomes via validated algorithms, but a reviewer will demand the outcome algorithm's PPV/sensitivity in that population, not a borrowed estimate. In elderly cohorts, differential competing risks by exposure (death competing with the event of interest, at different rates across arms) must be addressed with a pre-specified cause-specific vs cumulative-incidence decision, or the readiness reviewer will reject a naive Kaplan–Meier. - EHR: Strong for baseline severity (labs, vitals, problem lists, notes) but capture is visit-driven, so a patient who seeks care outside the system is differentially lost; readiness requires explicit observation windows, a missing-data pattern table, and linkage to dispensing/death where possible. The order/administration-vs-dispensing distinction must be reconciled before time-zero assignment. - Registry: Best for adjudicated outcomes and disease severity/stage, weakest for complete pharmacy exposure and mortality; readiness depends on documented linkage to claims (for fills) and a death index (for censoring), and on a transportability argument from the registry population to the decision population. - Linked claims–EHR–vital records: The ideal substrate (severity + completeness + reliable mortality) but linkage introduces selection (only the linkable subset) and date-discrepancy issues across order, fill, and service dates that must be reconciled and documented before time zero. A common, study-killing readiness defect here is immortal time in procedure-anchored designs: if follow-up starts at diagnosis but exposure is defined by a later procedure or fill, the interval guaranteeing survival to exposure is misclassified as exposed person-time — readiness forces time zero to the exposure decision and audits the timeline for it.

Worked example (claims, readiness gate-by-gate)

Question: does a second-generation sulfonylurea increase cardiovascular hospitalization vs a DPP-4 inhibitor in adults with type 2 diabetes, to support a post-marketing safety commitment, using a commercial + Medicare FFS database (2016–2023)? (1) Fit-for-purpose gate. Confirm the data can observe each estimand component: exposure (NDC fills with `days_supply`), the comparator (DPP-4 inhibitor fills), the outcome (CV hospitalization algorithm with documented PPV in this database), and complete person-time. Decision rule: require continuous medical + pharmacy enrollment and exclude MA-only person-time because FFS claims are absent there — without this, the washout is unverifiable. FAIL if the outcome algorithm has not been validated in a comparable population. (2) Estimand traceability gate (ICH E9(R1)). Lock the five attributes: population = incident users of either drug with ≥2 diabetes diagnoses and 365 days of continuous enrollment; treatments = sulfonylurea vs DPP-4 initiation; endpoint = first CV hospitalization; intercurrent events = death (competing risk → pre-specify cause-specific hazard and cumulative incidence), switching/discontinuation (treatment-policy vs while-on-treatment strategies named explicitly); summary = hazard ratio + 5-year cumulative incidence difference. Build a traceability matrix mapping each attribute to a SAP section and to a code module. (3) Time-zero / comparator defensibility gate. Time zero = first qualifying fill (NDC dispensed that day assigns the arm); washout = no fill of either class in the prior 365 days, so both arms are incident users and no immortal time is introduced. Defend the active comparator (both treat the same indication at the same decision point); show baseline covariate balance after the planned propensity-score adjustment using only pre-time-zero covariates. FAIL if the comparator is preferentially channeled (e.g., reserved for renal-impaired patients) without a documented balancing plan. (4) Sensitivity / falsification gate. Pre-specify negative-control outcomes and a negative-control exposure for empirical calibration, an E-value for the minimum unmeasured confounding that would explain the result, washout-length and grace-period variations, and a quantitative bias analysis for outcome misclassification. A readiness reviewer treats the absence of pre-specified falsification as a finding. (5) Transparency / reproducibility gate. Deliver the locked protocol/SAP (STaRT-RWE/HARPER fields complete), the attrition funnel from source population to analytic cohort, the full code with a data dictionary, and a versioned parameter table (`WASHOUT_DAYS=365`, grace period, caliper). The package PASSES readiness only when an independent analyst could re-derive the headline estimate from these artifacts.

Worked example

Scenario

A research team is planning an RWE study using commercial claims data to ask whether a second-generation sulfonylurea raises the risk of cardiovascular hospitalization compared with a DPP-4 inhibitor in adults with type 2 diabetes. The study will be submitted to the FDA as part of a post-marketing safety commitment. Before any code is written, the team must evaluate whether the study is regulatory-ready. The checklist below rates each required element as MET or UNMET and explains why it matters for the submission.

Dataset

Regulatory-readiness checklist for a cardiovascular safety study in T2D using commercial + Medicare FFS claims (2016-2023). Each row is one required element; the Status column shows whether this team has met it.

Steps

• Gate 1 (Fit-for-purpose data): Three of four elements pass. The blocking gap is that the outcome algorithm's accuracy has only been validated in Medicare, not in the commercial sub-population that makes up roughly half the study population. A reviewer will not accept a borrowed PPV.

• Gate 2 (Estimand traceability): The five ICH E9(R1) attributes are locked in the protocol, which is the harder half. But the traceability matrix — the document mapping each attribute to a SAP section and a code module — has not been written. Without it, an independent analyst cannot verify end-to-end that the code measures what the protocol claims.

• Gate 3 (Design defensibility): Both elements pass. Time zero is correctly set at first qualifying fill (no immortal-time risk), and the active comparator is justified as treating the same indication.

• Gate 4 (Sensitivity and falsification): One of two elements passes. The washout-length sensitivity analysis is pre-specified, but the negative-control outcome selection is absent. Reviewers flag missing negative controls as a readiness finding because they are the primary empirical check on residual confounding.

• Gate 5 (Transparency): Two of three elements pass. The blocking gap is the parameter table: the washout length, grace period, and propensity-score caliper are implemented in code but not documented in a versioned table that maps each value to its protocol justification.

• Overall verdict: 3 of 5 gates pass fully; 2 gates have at least one blocking gap. The study is NOT submission-ready in its current state. The three specific gaps — commercial-population PPV documentation, the estimand traceability matrix, and the versioned parameter table — must be resolved before data lock.

Result

NOT submission-ready. 9 of 12 checklist elements are MET; 3 are UNMET (outcome-algorithm PPV not validated in the commercial sub-population; estimand-to-code traceability matrix not written; versioned parameter table not finalized). All three gaps are fixable before data lock and do not require redesigning the study — but a submission package with any of these three open would receive a deficiency letter from a regulatory reviewer.

Runnable example

from dataclasses import dataclass, field

# A "gate" is a required readiness condition; each maps to evidence the protocol/SAP must contain.
GATES = {
    "fit_for_purpose": [
        "data_observes_exposure", "data_observes_comparator",
        "outcome_algorithm_ppv_documented", "person_time_observable",  # e.g., MA-only excluded
    ],
    "estimand_traceability": [
        "population_defined", "treatment_strategies_defined", "endpoint_defined",
        "intercurrent_events_strategy", "summary_measure_defined",
        "estimand_to_code_matrix",  # ICH E9(R1) attributes mapped to SAP + code modules
    ],
    "design_defensibility": [
        "time_zero_avoids_immortal_time", "comparator_justified",
        "baseline_covariates_pre_time_zero", "balance_plan_specified",
    ],
    "sensitivity_falsification": [
        "negative_controls_prespecified", "evalue_or_qba_planned",
        "washout_grace_sensitivity_planned",
    ],
    "transparency": [
        "protocol_sap_locked", "attrition_funnel", "code_and_data_dictionary",
        "parameter_table_versioned",  # WASHOUT_DAYS, grace period, caliper, etc.
    ],
}

@dataclass
class ReadinessResult:
    passed: bool
    gate_status: dict = field(default_factory=dict)   # gate -> bool
    blocking_gaps: list = field(default_factory=list) # "gate.requirement" strings

def assess_readiness(study: dict) -> ReadinessResult:
    """`study` keys are requirement names mapping to truthy (met) / falsy (missing or unmet)."""
    gate_status, gaps = {}, []
    for gate, requirements in GATES.items():
        missing = [r for r in requirements if not study.get(r)]
        gate_status[gate] = (len(missing) == 0)
        gaps.extend(f"{gate}.{r}" for r in missing)
    return ReadinessResult(passed=all(gate_status.values()),
                           gate_status=gate_status, blocking_gaps=gaps)

GATES <- list(
  fit_for_purpose = c("data_observes_exposure", "data_observes_comparator",
                      "outcome_algorithm_ppv_documented", "person_time_observable"),
  estimand_traceability = c("population_defined", "treatment_strategies_defined",
                            "endpoint_defined", "intercurrent_events_strategy",
                            "summary_measure_defined", "estimand_to_code_matrix"),
  design_defensibility = c("time_zero_avoids_immortal_time", "comparator_justified",
                           "baseline_covariates_pre_time_zero", "balance_plan_specified"),
  sensitivity_falsification = c("negative_controls_prespecified", "evalue_or_qba_planned",
                                "washout_grace_sensitivity_planned"),
  transparency = c("protocol_sap_locked", "attrition_funnel",
                   "code_and_data_dictionary", "parameter_table_versioned")
)

assess_readiness <- function(study) {
  # study: named logical vector; TRUE = requirement met. Missing names default to FALSE.
  met <- function(req) isTRUE(study[[req]])
  gate_status <- sapply(GATES, function(reqs) all(vapply(reqs, met, logical(1))))
  gaps <- unlist(lapply(names(GATES), function(g)
    paste0(g, ".", GATES[[g]][!vapply(GATES[[g]], met, logical(1))])))
  list(passed = all(gate_status), gate_status = gate_status, blocking_gaps = gaps)
}