Estimand-to-Analysis Traceability
A documentation discipline that maps each attribute of a pre-specified estimand (treatment, population, endpoint, intercurrent-event strategy, summary measure) to the exact operational data rule, code module, diagnostic, and sensitivity analysis that implements it, so the number produced answers the causal question that was asked.
In plain language
Before running a single line of code, a researcher writes out the precise question they want to answer — who the patients are, which treatments are compared, what outcome counts, what happens when patients switch drugs mid-study, and how to summarize the result. That written question is called the target question, and traceability means every analysis choice made afterward can be traced back to one of those five parts of the question, so that the final number provably answers the question that was asked. Without this discipline, it is easy to drift: the code quietly answers a slightly different question — for example, treating drug-switching as if it never happened rather than deciding in advance what to do about it. Traceability makes that drift visible before results are shared.
Estimand-to-analysis traceability
is the connective tissue between a causal question and the number a study reports. The ICH E9(R1) estimand framework decomposes the target quantity into five attributes — treatment conditions, population, endpoint (variable), intercurrent-event (ICE) strategy, and population-level summary measure. Traceability is the requirement that every one of those attributes be carried, without slippage, through cohort construction, follow-up, outcome ascertainment, the statistical model, and the sensitivity plan — and that the mapping be written down (a traceability matrix) so an author, reviewer, or regulator can audit "the estimate answers the estimand." It is not itself a study design or an estimator; it is the governance layer that prevents the silent substitution of a different estimand that observational data make convenient. In RWE this discipline matters more than in trials, because the data were collected for billing or care, not for the question, so the gap between "what we wanted to estimate" and "what the code actually computed" is wider and easier to cross unnoticed.
Core conceptual distinction
. The thing to keep separate is the estimand (the target quantity, defined before any data are touched) from the estimator (the statistical procedure) and the estimate (the realized number). Traceability is the bookkeeping that binds the three so they refer to the same target. The most consequential RWE failures are estimand drift: follow-up that starts at diagnosis rather than at the treatment decision silently changes the population attribute (it admits immortal person-time); censoring at treatment discontinuation without weighting changes the ICE strategy from a treatment-policy to a hypothetical/while-on-treatment estimand; reporting a hazard ratio when the question is about absolute risk in the presence of competing death changes the summary measure from a cumulative-incidence contrast to a cause-specific rate that no patient experiences. Each is a defensible analysis of some estimand — just not the one the protocol named. Traceability makes the substitution visible by forcing the matrix cell "ICE strategy = treatment policy" to point at the specific code that does NOT censor at switching, and the diagnostic that proves it.
Pros, cons, and trade-offs
. - vs an undocumented single-rule implementation (the common default — write the SAS, get a number): traceability is far more transparent, reproducible, and defensible for FDA/EMA/HTA submission, and it catches estimand drift before it reaches a reviewer. Cost: it is overhead — a matrix to build and maintain, more diagnostics, more sensitivity runs. Prefer it for any consequential comparative-effectiveness, safety, utilization, cost, or regulatory-grade analysis. - vs the target-trial-emulation protocol alone: a target-trial protocol is an estimand specification (eligibility, treatment strategies, assignment, follow-up, outcome, causal contrast). Traceability is the layer beneath it that maps each protocol element to executable code and a check. The protocol says "treatment-policy estimand"; the traceability matrix proves the code implemented it. Use both — the protocol is the spine, the matrix is the audit trail. They are complements, not substitutes. - vs leaving estimand choice implicit in the SAP's model section: putting the estimand only in the model paragraph invites the reader to reverse-engineer the question from the procedure (PROC PHREG ⇒ "they must want a hazard ratio"), which is exactly backwards and hides ICE handling. A standalone matrix forces the question first. Cost: redundancy with the SAP; resolve by making the matrix a SAP appendix rather than a parallel document.
When to use
. Any hypothesis-evaluating RWE study intended for a regulator, payer, or HTA body; any study where the estimand has a non-trivial ICE strategy (treatment switching, discontinuation, death as a competing event, rescue medication); any multi-programmer or multi-database study where drift between sites is plausible; and as a precondition for pre-registration, because you cannot pre-specify what you cannot trace. Build the matrix at protocol/SAP time, before programming, and freeze it with the analysis plan.
When NOT to use — and when it is actively misleading or dangerous
. For an exploratory, hypothesis-generating descriptive scan with no single target quantity, a full estimand matrix is ceremony that buys nothing — say so rather than manufacture a fake estimand. The dangerous failure mode is traceability theater: a matrix that is filled in after the analysis to rationalize whatever the code happened to compute. A post-hoc matrix that maps "ICE = treatment policy" to code that in fact censored at switching is worse than no matrix — it launders estimand drift with the appearance of rigor and will mislead a reviewer who trusts the document. The discipline is only protective if the matrix is written before the code and the diagnostics are run to confirm each cell, not to decorate it. Equally, a matrix that is internally consistent but built on an estimand that the data cannot support (e.g., a while-on-treatment estimand in a claims source where treatment episodes cannot be reconstructed from `days_supply`) gives false comfort: traceability guarantees the estimate matches the estimand, not that the estimand is identifiable from the source.
Data-source operational depth
. Traceability is where data-source pathology becomes estimand-specific, because each attribute fails differently by source. - Claims (FFS vs MA vs commercial): The population and follow-up attributes hinge on continuous, observable enrollment. Medicare Advantage encounter data lack the fee-for-service claims used to ascertain exposure and outcomes, so MA-only person-time is missingness, not absence — a "treatment policy from first fill" estimand silently becomes "treatment policy among the FFS-observable," a different population. Fix: require continuous Parts A/B/D (or commercial medical+pharmacy) across the window and exclude MA-only spans, and record that exclusion in the population cell. The treatment attribute (treatment policy vs while-on-treatment) is operationalized through `days_supply` + a grace period; sample fills, 90-day mail order, and free samples corrupt `days_supply` and quietly shift a while-on-treatment estimand. The endpoint attribute for any elderly/comorbid cohort must contend with differential competing risks — if death rates differ by arm, a cause-specific Cox HR and a Fine-Gray subdistribution contrast diverge, and only one matches a "cumulative incidence" summary measure. - EHR: The endpoint and follow-up attributes are encounter-driven; a patient who leaves the system is differentially lost, so a "treatment policy through end of follow-up" estimand requires an explicit observation-window definition and informative-censoring handling, not the implicit "last seen" that the data offer. The treatment attribute starts at the order/administration, not the dispense — trace which. - Registry / linked: Registries are strong for the endpoint (adjudicated events, disease stage) and population (severity) but weak for treatment (incomplete pharmacy). Linkage to claims fills the treatment cell and to a death index firms up the competing-risk and censoring cells — but linkage selection narrows the population attribute to the linkable subset, which the matrix must state. Immortal-time in procedure/device studies is an estimand-population failure: if eligibility requires surviving to a procedure date, the matrix's "time zero" cell must align follow-up start with the treatment decision, or the population silently excludes early deaths.
Worked example (claims traceability matrix)
Question: comparative risk of hospitalization for heart failure (HHF) with an SGLT2 inhibitor vs a DPP-4 inhibitor among adults with type 2 diabetes in a Medicare FFS + commercial database; target estimand = the treatment-policy effect on cumulative incidence of HHF at 2 years, accounting for death as a competing event. The matrix binds each ICH E9(R1) attribute to one operational rule, one code module, one diagnostic, and one pre-specified sensitivity: (1) Treatment = initiation of SGLT2i vs DPP-4i, treatment-policy strategy → arm assigned from the NDC on the first qualifying `fill_date`; the as-treated alternative would stitch `days_supply` into episodes with a 30-day grace period — code module `build_episodes()`; diagnostic = distribution of `days_supply` and gap lengths by arm; sensitivity = re-run as while-on-treatment with IPC weighting. (2) Population = incident users with T2D and 365 days of continuous Parts A/B/D (no MA-only span) → `apply_enrollment()`; diagnostic = attrition funnel and % MA-only excluded; sensitivity = relax washout to 180 days. (3) Endpoint = first HHF (validated 1-inpatient HF claim in the primary position) → `flag_hhf()`; diagnostic = PPV against a chart-validated subset; sensitivity = broaden to any-position HF. (4) Intercurrent events = death is a competing event (handled by the subdistribution model, NOT censored), switching handled by the treatment-policy strategy (ignored) → `set_competing_risk()`; diagnostic = cumulative-incidence-function plot of HHF and death by arm; sensitivity = cause-specific Cox to show the gap. (5) Summary measure = 2-year cumulative-incidence difference and ratio with a Fine-Gray subdistribution hazard ratio after overlap weighting → `fit_finegray()` (e.g., `PROC PHREG ... eventcode=` / `survival::finegray`); diagnostic = standardized mean differences <0.1 post-weighting; sensitivity = E-value for the SHR. Every cell that a reviewer might suspect of drift — "did they censor at switching?" "did they treat death as censoring?" — resolves to a named module and a check, which is the entire point of the matrix.
Worked example
Scenario
A researcher wants to know: among adults newly starting an SGLT2 inhibitor versus a DPP-4 inhibitor for type 2 diabetes in Medicare claims, what is the difference in two-year cumulative risk of hospitalization for heart failure, treating death as a competing event and ignoring any later drug switching? Before touching data, they write this question out as five attributes and then trace each attribute to a specific analysis choice. The table below shows that trace. No numbers are fabricated here — the point is the logic of the mapping, not a specific effect size.
Dataset
Traceability matrix: five attributes of the pre-specified target question, each mapped to one operational rule and one analysis choice. This table is written before any code is run.
| Attribute | What the attribute pins down | Operational rule written in the analysis plan | Analysis choice it drives |
|---|---|---|---|
| Population | Who counts as a study patient | Adults with type 2 diabetes who fill their first-ever SGLT2i or DPP-4i prescription after 365 days of continuous insurance enrollment, with no prior fill of either drug class | The cohort-entry code requires a 365-day clean lookback and a confirmed diabetes diagnosis before the first fill; patients already on either drug are excluded |
| Treatment | Which exposures are compared and how they are defined | Arm assigned by the drug class on the index fill date; later switching is ignored (treatment-policy approach — the patient stays in the arm they started in) | The code reads the NDC code on the first qualifying fill and locks the arm; no re-assignment logic runs when switching occurs in follow-up |
| Endpoint | What outcome is measured and how it is recognized in the data | First inpatient hospitalization with heart failure coded in the primary diagnosis position; the patient must be admitted, not just seen in the ER | The outcome flag requires one inpatient claim with a qualifying ICD-10 code in position 1; ER visits and observation stays do not trigger the flag |
| Handling of intercurrent events | What to do when a patient switches drugs or dies before a heart-failure hospitalization occurs | Death is a competing event and is NOT treated as ordinary censoring; switching is handled by the treatment-policy approach above (stay in original arm through the full two-year window) | The analysis uses a competing-risks model so that the risk of heart failure and the risk of death are estimated simultaneously; patients who die are not simply dropped from the denominator |
| Summary measure | How the result is expressed as a single comparable number | Two-year cumulative incidence difference and ratio between arms, estimated after balancing the two groups on measured confounders using overlap weighting | The model reports absolute risk at two years for each arm and their difference; it does not report a hazard ratio, because the pre-specified question is about cumulative risk, not instantaneous rate |
Steps
Write the five attributes before opening any data file. Each attribute locks one dimension of the question; leaving any attribute unspecified means the code will make that choice invisibly.
For each attribute, write the operational rule in plain language specific enough that two programmers would produce the same result. Vague rules such as 'new users' or 'first fill' are not operational until the exact lookback period and exclusion criteria are stated.
For each attribute, identify which analysis choice it forces. The population attribute determines the cohort-entry query; the intercurrent-event attribute determines whether the outcome model censors at death or treats it as a competing event. These are not independent decisions — they are forced by the pre-specified question.
Lock the table before writing code. After the analysis runs, check each row: does the code actually implement the operational rule? A row where the code diverges from the rule signals estimand drift — fix the code, not the question.
Run the pre-specified sensitivity analyses for any attribute that involved a judgment call. For example, re-run the population attribute with a 180-day lookback to verify results do not change materially; re-run the intercurrent-event attribute using a cause-specific Cox model to show how much the competing-risks handling matters.
Result
The completed matrix produces a fully specified question: the two-year absolute risk difference in heart-failure hospitalization between new SGLT2i and DPP-4i users in Medicare, among incident users with 365 days of prior enrollment, assigning arms at the index fill, treating death as a competing event, estimated as a cumulative-incidence contrast after overlap weighting. Every word in that sentence maps to one row of the table. A reviewer can audit the number by checking whether the code implements each row — that auditability is the entire purpose of the discipline.
Runnable example
python implementation
Programmable estimand traceability matrix. This is a framework artifact, not an estimator: it encodes — as data — the binding between each ICH E9(R1) estimand attribute and the operational rule, code module, diagnostic, and sensitivity analysis that...
import pandas as pd
# One row per ICH E9(R1) estimand attribute. Fill from the protocol/SAP BEFORE programming.
# Worked example: treatment-policy effect on 2-year cumulative incidence of HF hospitalization,
# SGLT2i vs DPP-4i, Medicare FFS + commercial claims, death as a competing event.
REQUIRED_ATTRIBUTES = [
"treatment", "population", "endpoint", "intercurrent_events", "summary_measure",
]
matrix = pd.DataFrame([
{"attribute": "treatment",
"operational_rule": "Arm = NDC on first qualifying fill_date; treatment-policy (ignore later switching)",
"module": "build_cohort.assign_arm",
"diagnostic": "days_supply + gap-length distribution by arm",
"sensitivity": "as-treated: build_episodes(grace_days=30) + IPC weighting"},
{"attribute": "population",
"operational_rule": "Incident T2D users, 365d continuous Parts A/B/D, exclude MA-only person-time",
"module": "build_cohort.apply_enrollment",
"diagnostic": "attrition funnel; % excluded for MA-only / non-continuous",
"sensitivity": "washout 180d vs 365d"},
{"attribute": "endpoint",
"operational_rule": "First HF hospitalization: 1 inpatient HF claim in primary position",
"module": "outcomes.flag_hhf",
"diagnostic": "PPV vs chart-validated subset",
"sensitivity": "any-position HF dx"},
{"attribute": "intercurrent_events",
"operational_rule": "Death = competing event (NOT censored); switching = ignored (treatment policy)",
"module": "outcomes.set_competing_risk",
"diagnostic": "cumulative-incidence-function plot of HF and death by arm",
"sensitivity": "cause-specific Cox to quantify the divergence"},
{"attribute": "summary_measure",
"operational_rule": "2y cumulative-incidence difference + Fine-Gray subdistribution HR, overlap-weighted",
"module": "analysis.fit_finegray", # e.g. survival::finegray / PROC PHREG eventcode=
"diagnostic": "standardized mean differences < 0.1 post-weighting",
"sensitivity": "E-value for the subdistribution HR"},
])
def validate_traceability(m: pd.DataFrame) -> list[str]:
"""Return audit failures: untraced attributes or empty implementation cells."""
problems = []
missing = set(REQUIRED_ATTRIBUTES) - set(m["attribute"])
if missing:
problems.append(f"estimand attributes with no row (untraced): {sorted(missing)}")
for _, row in m.iterrows():
for col in ("operational_rule", "module", "diagnostic", "sensitivity"):
if not str(row.get(col, "")).strip():
problems.append(f"attribute '{row['attribute']}' has empty {col}")
# The intercurrent-event cell is the highest-drift risk; require it to name a strategy.
ice = m.loc[m["attribute"] == "intercurrent_events", "operational_rule"]
strategies = ("treatment policy", "while-on-treatment", "hypothetical", "composite", "principal")
if ice.empty or not any(s in ice.iloc[0].lower() for s in strategies):
problems.append("intercurrent_events row does not name an ICH E9(R1) ICE strategy")
return problems
issues = validate_traceability(matrix)
assert not issues, "Traceability matrix failed audit:\n" + "\n".join(issues)r implementation
Estimand traceability matrix as a tidy data frame plus an auditor. Mirrors the Python artifact: each ICH E9(R1) attribute is bound to its operational rule, the analysis module/PROC that implements it, the confirming diagnostic, and the pre-specified...
library(tibble)
required_attributes <- c("treatment", "population", "endpoint",
"intercurrent_events", "summary_measure")
# Worked example: treatment-policy effect on 2-year cumulative incidence of HF hospitalization,
# SGLT2i vs DPP-4i, Medicare FFS + commercial claims, death as a competing event.
matrix <- tibble::tribble(
~attribute, ~operational_rule, ~module, ~diagnostic, ~sensitivity,
"treatment", "Arm = NDC on first qualifying fill_date; treatment-policy (ignore switching)", "assign_arm", "days_supply + gap-length distribution by arm","as-treated: build_episodes(grace=30) + IPC weights",
"population", "Incident T2D users, 365d continuous A/B/D, exclude MA-only person-time", "apply_enrollment", "attrition funnel; % MA-only excluded", "washout 180d vs 365d",
"endpoint", "First HF hospitalization: 1 inpatient HF claim in primary position", "flag_hhf", "PPV vs chart-validated subset", "any-position HF dx",
"intercurrent_events", "Death = competing event (NOT censored); switching ignored (treatment policy)", "set_competing_risk", "CIF plot of HF and death by arm", "cause-specific Cox to quantify divergence",
"summary_measure", "2y cumulative-incidence difference + Fine-Gray subdistribution HR, overlap-wtd", "fit_finegray", "standardized mean differences < 0.1", "E-value for subdistribution HR"
)
validate_traceability <- function(m) {
problems <- character(0)
missing <- setdiff(required_attributes, m$attribute)
if (length(missing) > 0)
problems <- c(problems, paste("untraced estimand attributes:", paste(missing, collapse = ", ")))
for (col in c("operational_rule", "module", "diagnostic", "sensitivity")) {
empty <- m$attribute[!nzchar(trimws(m[[col]]))]
if (length(empty) > 0)
problems <- c(problems, paste0("empty ", col, " for: ", paste(empty, collapse = ", ")))
}
ice <- m$operational_rule[m$attribute == "intercurrent_events"]
strategies <- c("treatment policy", "while-on-treatment", "hypothetical", "composite", "principal")
if (length(ice) == 0 || !any(vapply(strategies, function(s) grepl(s, tolower(ice)), logical(1))))
problems <- c(problems, "intercurrent_events row names no ICH E9(R1) ICE strategy")
problems
}
issues <- validate_traceability(matrix)
if (length(issues) > 0) stop("Traceability matrix failed audit:\n", paste(issues, collapse = "\n"))