Study Protocol and SAP Elements for RWE

In plain language

A study protocol and statistical analysis plan (SAP) are documents you write before looking at results that spell out every decision about your study: who is in it, what counts as the treatment, what counts as the outcome, how long you follow people, and exactly how you will analyze the data. Pre-specifying these choices prevents a researcher from, knowingly or not, tweaking definitions until the answer looks good. Think of it as writing your hypothesis on a whiteboard and taking a photo before the test, so no one can accuse you of moving the target afterward.

A real-world-evidence (RWE) study protocol and its companion statistical analysis plan (SAP) are the pre-specification layer that turns a research question into a reproducible, auditable analysis. They are not paperwork: each element — the data source and its observable person-time, eligibility, time zero, exposure and comparator definitions, outcome algorithm, follow-up and censoring rules, covariates and their measurement window, the estimand, the primary analysis, and the pre-specified sensitivity analyses — is a design decision that changes who is in the cohort, how long they are followed, how the outcome is counted, and what the effect estimate means. Structured templates (STaRT-RWE, HARPER) exist precisely because the failure mode of RWE is not a single wrong number but an undocumented chain of analyst choices that no reviewer can reconstruct. The discipline is to write every element in protocol language, lock it (ideally with a timestamp before outcome data are touched, or a pre-registration), and only then program.

Core conceptual distinction

A protocol/SAP is specification, not estimation. Three boundaries do the work and are routinely confused. (1) Protocol vs SAP: the protocol fixes the design (PICOT-style question, data source, eligibility, time zero, exposure/comparator, outcome, follow-up); the SAP fixes the analytic detail (estimator, model form, covariate list and functional form, missing-data and censoring handling, multiplicity, the exact sensitivity analyses). (2) Design-stage vs analysis-stage decisions: time zero, washout, and the eligibility/exposure windows are design decisions that must be set without reference to outcomes; choosing them after looking at results is the archetype of specification-search bias. (3) Pre-specified vs data-driven: a sensitivity analysis named in the SAP is evidence; the same analysis run because the primary result looked wrong is a post-hoc rationalization. The unifying object is the estimand — the population, treatment contrast, endpoint, intercurrent-event strategy, and summary measure — which the protocol must state explicitly (ICH E9(R1) language) so that the SAP's estimator and the reviewer's interpretation refer to the same target quantity.

Pros, cons, and trade-offs

- vs an ad hoc analysis script with no protocol: A locked protocol/SAP makes the study reproducible and defensible, exposes design decisions to review before they are contaminated by results, and is now an explicit expectation of FDA, EMA, and HTA bodies. Cost: real up-front effort and reduced freedom to chase interesting post-hoc findings. Prefer the protocol for any decision-grade comparative, safety, utilization, or economic analysis — i.e., almost always. - vs a clinical-trial protocol/SAP template reused unchanged: Trial templates assume randomization, a clean enrollment visit, and protocol-defined assessments. RWE templates (STaRT-RWE, HARPER) add what trials get for free and RWE must engineer: an observable-person-time definition for the data source, an explicit time-zero rule, code lists with provenance and validation metrics, and confounding control. Prefer an RWE-specific template; bolting RWE onto an ICH E6 trial protocol silently omits the elements that drive RWE bias. - vs lightweight pre-registration (e.g., a one-paragraph hypothesis registry): Pre-registration deters outcome switching but does not specify the operational rules (washout, days_supply stitching, censoring) that determine the estimate. Use both: register to fix the question and timestamp, and a full STaRT-RWE/HARPER protocol+SAP to fix the operations. Registration without operational pre-specification gives a false sense of rigor. - vs maximal pre-specification of every contingency: Over-specifying can lock in a wrong model when the data reveal an unanticipated structure (e.g., non-proportional hazards). The trade-off is resolved by pre-specifying the primary analysis rigidly and pre-specifying named, triggered sensitivity/secondary analyses with their decision rules, rather than leaving the analyst free hands.

When to use

Any hypothesis-evaluating RWE study intended to inform a regulatory submission, label expansion, safety signal evaluation, HTA dossier, payer coverage decision, or peer-reviewed comparative-effectiveness claim. Use a structured template (STaRT-RWE for the analysis-implementation table; HARPER for the full protocol narrative) whenever a reviewer, regulator, or replication team will need to reconstruct the study, and whenever the analysis involves a non-trivial time-zero, washout, exposure-episode, or censoring rule — i.e., whenever the design itself can introduce immortal time, selection, or confounding.

When NOT to use — and when it is actively misleading or dangerous

- Pure hypothesis-generating description. For exploratory feasibility counts or initial data profiling, a heavy locked SAP adds friction without protecting a decision; a lightweight analysis plan suffices. Forcing a full confirmatory protocol on exploration can dress data-dredging up as confirmatory evidence — the opposite of the intent. - A protocol written after the analysis, to match the result. A post-hoc protocol that reverse-engineers the chosen design is worse than none: it manufactures the appearance of pre-specification and is the single most dangerous failure mode in RWE submissions. If outcomes have been examined, say so and treat the work as hypothesis-generating. - Specification so vague it cannot be programmed two ways the same. "Continuous enrollment around index" or "relevant comorbidities adjusted for" is non-pre-specification masquerading as a protocol; it leaves every consequential choice to undocumented analyst discretion and is not auditable. - Estimand left implicit. If the protocol does not state whether the target is an ITT-like initiation contrast or an as-treated/per-protocol contrast (and how intercurrent events — switching, discontinuation, death — are handled), the SAP's estimator and the conclusion can disagree without anyone noticing; the "result" is then uninterpretable rather than merely imprecise.

Data-source operational depth

The protocol's observable-person-time definition is the element most often underspecified, and it differs sharply by source. - Claims (FFS vs Medicare Advantage): Person-time is observable only while the enrollee contributes the relevant benefit. Medicare Advantage encounter data historically lack complete fee-for-service (FFS) claims, so an MA-only interval can read as "no events / no fills" when it is really unobserved — the protocol must restrict to FFS Parts A/B/D (or commercial medical+pharmacy) across washout and follow-up and explicitly exclude MA-only person-time, or it will manufacture immortal-like gaps and undercount events. Specify continuous-enrollment and allowable-gap rules, `days_supply`-based exposure-episode construction (with mail-order 90-day and sample-fill caveats), claims-adjudication lag and run-out windows, and reversal handling — each in the SAP, not left to the programmer. - EHR: Capture is encounter-driven, so person-time is observable only when the patient is active in the network; a patient who seeks care elsewhere is differentially lost. The protocol must define observation windows from encounter density (not enrollment), pre-specify how external-care leakage and missing structured fields are handled, and — for elderly or sicker cohorts — pre-specify a competing-risks framing, because death and other terminal events occur differentially by exposure and a naive censoring rule biases cumulative incidence. - Registry: Strong for adjudicated outcomes and disease severity, weak for complete exposure and for vital status; the protocol must pre-specify linkage to claims for fills and to a death index for censoring, and state the registry completeness/adjudication rules and the eligible-for-linkage denominator. - Linked claims–EHR–vital records: The richest substrate, but linkage selects the linkable subset and creates order/fill/service date discrepancies that must be reconciled before time zero is assigned; the protocol must pre-specify the date-reconciliation rule and report the linkage-eligible vs analyzed denominators.

Two cross-cutting failure modes belong in every RWE protocol because they are created by under-specification: immortal time (e.g., in procedure or "responder" studies where follow-up starts before the exposure-defining event — the protocol must set time zero at the exposure decision, not at diagnosis or at procedure completion), and differential competing risks by exposure in elderly claims cohorts (handle with cause-specific or subdistribution models, pre-specified, not chosen after seeing the curves).

Worked claims example (protocol elements, enumerated)

Question: among Medicare FFS beneficiaries with non-valvular atrial fibrillation, does initiation of a direct oral anticoagulant (DOAC) vs warfarin change the rate of hospitalized major bleeding? (1) Data source / observable person-time: Medicare Parts A/B/D FFS; require continuous A/B/D and exclude any Medicare Advantage interval (MA encounter data lack complete FFS claims, so MA person-time is unobserved). (2) Eligibility: age ≥65, ≥1 inpatient or ≥2 outpatient AF diagnoses (`dx` in baseline), and 365 days of continuous FFS enrollment before the first qualifying fill. (3) Time zero: date of the first DOAC or warfarin pharmacy claim (`fill_date`); assign the arm from the NDC dispensed that day — set follow-up at the fill, not at AF diagnosis, to avoid immortal time. (4) Washout / new-user: no DOAC or warfarin fill in the 365-day lookback, so both arms are incident users. (5) Outcome: first hospitalized major bleeding via a validated claims algorithm (specify ICD position and setting; cite PPV/sensitivity). (6) Follow-up and censoring: from time zero to outcome, disenrollment, end of data, or — as-treated — last `days_supply` end plus a pre-specified grace period or switch; death is a competing risk, so the SAP pre-specifies a Fine–Gray subdistribution model alongside a cause-specific sensitivity analysis rather than censoring deaths. (7) Covariates: measured only in the 365-day baseline window (CHA₂DS₂-VASc and HAS-BLED components, prior bleeds, renal disease, concomitant antiplatelets, utilization), feeding a high-dimensional propensity score. (8) Estimand: the as-treated comparative subdistribution hazard of major bleeding for DOAC vs warfarin initiation, with switching/discontinuation handled by the censoring + IPCW strategy stated above (ICH E9(R1) intercurrent-event framing). (9) Primary analysis: 1:1 PS-matched Fine–Gray model; balance confirmed by standardized differences <0.1. (10) Pre-specified sensitivity analyses: washout length (180 vs 365 d), grace period, cause-specific vs subdistribution hazard, a negative-control outcome to detect residual confounding, and restriction to overlapping calendar time. Locking all ten elements before pulling outcome data is what makes the eventual estimate interpretable and defensible.

Worked example

Scenario

A researcher wants to know whether patients newly started on Drug A have fewer hospitalizations over the following year than patients newly started on Drug B. Before touching outcome data, the team locks a protocol and SAP. The table below shows the eight core protocol and SAP elements they must pre-specify, what each element says for this concrete question, and why locking that element ahead of time prevents a specific form of fishing.

Dataset

Core protocol and SAP elements for a comparative-effectiveness study of Drug A vs Drug B and hospitalization, showing each element, its pre-specified value, and the bias it prevents.

Steps

• Draft the objective first so the whole team agrees on exactly one question; any ambiguity here will cascade into ambiguous exposure and outcome definitions later.

• Choose the data source and study period without looking at outcome rates; changing the database after seeing results is a form of selection bias.

• Write the eligibility rules, including the lookback washout length, before running any cohort counts; the washout length directly affects who is a new user and therefore what the effect estimate means.

• Lock the exposure NDC code lists with a version date so a reviewer can reproduce exactly who was assigned to each arm.

• Specify the outcome algorithm with its validation metric (PPV) before unblinding; a high-PPV algorithm and a low-PPV algorithm will give different event counts, and choosing after seeing results is outcome switching.

• List every covariate and state that measurement is restricted to the pre-index baseline window; any covariate measured after index date could itself be affected by treatment and would distort the adjustment.

• Name the primary analysis method in full detail; if you specify three plausible models in the SAP and commit to reporting only the pre-specified primary, you cannot later cherry-pick the one with the smallest p-value.

• Write the sensitivity analyses with explicit trigger conditions before unblinding; a sensitivity analysis that appears only because the main result looked wrong is a post-hoc rationalization, not confirmatory evidence.

Result

When all eight elements are locked before the first outcome-dependent query, the final hazard ratio is interpretable and auditable: a reviewer can reconstruct every analytic choice, confirm they were made without knowledge of the result, and trust that the estimate answers the stated question rather than the question that happened to yield a favorable number.

Runnable example

# protocol_elements.yaml  --  lock BEFORE touching outcome data
question:
  pico_t: "Among Medicare FFS adults with non-valvular AF, does DOAC vs warfarin initiation
           change the rate of hospitalized major bleeding?"
data_source:
  name: "Medicare FFS Parts A/B/D"
  observable_person_time: "continuous A/B/D enrollment; EXCLUDE Medicare Advantage intervals (FFS claims absent)"
  study_period: "2016-01-01 .. 2022-12-31"
eligibility:
  age_min: 65
  indication: ">=1 inpatient OR >=2 outpatient AF dx (ICD-10 I48.x) in 365d baseline"
  enrollment: "365d continuous FFS A/B/D before first qualifying fill"
time_zero:
  rule: "date of first DOAC or warfarin pharmacy claim (fill_date)"   # NOT AF diagnosis -> avoids immortal time
  arm_assignment: "drug class of the NDC dispensed on fill_date"
washout_new_user:
  lookback_days: 365
  rule: "no DOAC or warfarin fill in lookback (both arms incident users)"
exposure:
  study: "DOAC NDC list (v2024-03, provenance: RED BOOK)"
  comparator: "warfarin NDC list (v2024-03)"
  episode: "days_supply stitching; grace_period_days: 14; switch = fill of other class"
outcome:
  definition: "first hospitalized major bleeding (validated algorithm; PPV 0.89 per Cunningham 2011)"
  coding: "primary/secondary ICD-10 in inpatient setting"
follow_up_censoring:
  start: "time_zero"
  end_at: ["outcome", "disenrollment", "end_of_data", "as_treated: last days_supply + grace OR switch"]
  death: "COMPETING RISK -> Fine-Gray subdistribution (primary); cause-specific (sensitivity)"  # do NOT censor deaths
covariates:
  window: "[time_zero - 365d, time_zero]"   # baseline only; never post-time-zero
  list: ["CHA2DS2-VASc components", "HAS-BLED components", "prior_bleed", "renal_disease",
         "concomitant_antiplatelet", "baseline_utilization"]
  model: "high-dimensional propensity score"
estimand:
  target: "as-treated comparative subdistribution hazard, DOAC vs warfarin initiation"
  intercurrent_events: "switching/discontinuation via censoring + IPCW; death as competing risk (ICH E9(R1))"
primary_analysis:
  method: "1:1 PS matching -> Fine-Gray model"
  balance_check: "standardized differences < 0.1 post-match"
sensitivity_analyses:   # pre-specified, with triggers -- not chosen after seeing results
  - "washout 180d vs 365d"
  - "grace period 7/14/30d"
  - "cause-specific vs subdistribution hazard"
  - "negative-control outcome (residual confounding probe)"
  - "restrict to overlapping calendar time of both drugs"
governance:
  locked_on: "YYYY-MM-DD"
  data_extract_date: "YYYY-MM-DD"
  registration: "ENCePP/ClinicalTrials.gov ID"