Fit-for-Purpose Data Assessment

In plain language

Before designing any real-world study, researchers must ask one question: does the database we are considering actually contain what we need to answer our specific question? Fit-for-purpose data assessment is the structured process that answers that question by checking two things: relevance (does the source capture the right patients, the right drug, the right outcome, and enough follow-up time?) and reliability (are those records accurate, complete, and produced by a trustworthy data process?). The output is a single, defensible verdict — go, no-go, or go only if certain gaps are addressed — tied to that one question, not to the database in general. A database can be perfectly fit for counting prescriptions and completely unfit for measuring whether patients die from a heart attack, even when both studies use the exact same patients.

Fit-for-purpose (FFP) data assessment

is the gatekeeping step that decides, before any analysis is programmed, whether a particular real-world data source can credibly answer a particular question. It is not a generic "data quality score" and it is not transferable: a database can be fit for purpose for a comparative drug-utilization study and entirely unfit for a comparative mortality study run on the same patients. The assessment is organized around two axes made canonical by the FDA RWD guidance and operationalized by Gatto et al.'s Structured Process to Identify Fit-for-Purpose Data (SPIFD): relevance — does the source contain the population, exposure, outcome, key confounders, and follow-up duration the estimand demands? — and reliability — are those elements accurate, complete, traceable, and produced by a stable, documented data-curation (ETL) process? The output is a documented go / no-go / go-with-mitigations verdict for one PICOTS-defined question, plus the specific sensitivity analyses that will probe the most judgment-dependent thresholds.

Core conceptual distinction

. FFP assessment sits upstream of, and is distinct from, three things it is often confused with. (1) vs database feasibility / attrition counting: a feasibility funnel tells you how many patients survive each eligibility step; FFP tells you whether the surviving cohort and its variables mean what the protocol needs them to mean. Feasibility is a necessary input to FFP, not a substitute. (2) vs algorithm/outcome validation: validation estimates the operating characteristics (PPV, sensitivity) of one variable; FFP integrates those characteristics with relevance and curation evidence into a question-level decision. (3) vs a global data-quality grade: question-agnostic grading (completeness %, conformance checks) is a reliability input but cannot, by itself, declare fitness, because fitness is defined relative to a specific estimand. The decisive output of FFP is therefore not a number but a defensible decision tied to a question, with the residual risks named and mitigated.

Pros, cons, and trade-offs

(specific & comparative, naming the alternatives). - vs proceeding straight to analysis on a convenient database: FFP forces relevance/reliability to be argued in protocol language before code is written, which is exactly what FDA and EMA reviewers expect and what prevents an expensive study from being rejected for an avoidable data limitation (e.g., MA-only person-time with no fee-for-service claims, no death linkage for a mortality endpoint). Cost: it is front-loaded work that delays the first results and requires data-provenance documentation the vendor may not readily supply. Prefer FFP for any regulatory-grade or HTA-facing study. - vs a one-time, question-agnostic data-quality scorecard: a reusable scorecard is cheap and comparable across databases, but it systematically over- or under-states fitness because it ignores the estimand — a source with 99% completeness on labs is still unfit for a question that turns on outpatient mortality. FFP is question-specific and therefore more defensible, at the price of being non-transferable and needing re-execution for each new question. Prefer the scorecard only as a pre-screen to shortlist databases, then run FFP on the finalists. - vs multi-database replication as the primary safeguard: running the analysis in several databases is powerful against database-specific artifacts, but it is reactive and expensive, and concordant-but-wrong results across sources that share a structural flaw (e.g., all lack reliable cause-of-death) give false reassurance. FFP is proactive and cheaper. Use both when stakes are high: FFP to qualify each source, replication to test robustness.

When to use

. Any regulatory submission (FDA RWE program, EMA), HTA dossier, or post-authorization safety/effectiveness study; the data-selection step of a target-trial emulation, where each trial component (eligibility, treatment strategies, assignment, outcome, follow-up, estimand) must be checked against what the source can actually capture; any high-consequence comparative-effectiveness, safety, utilization, or cost analysis where a data limitation could change the conclusion. Execute FFP once the PICOTS and estimand are fixed but before locking the analytic protocol, so that the verdict can still redirect the study to a different source, a narrower question, or a linkage strategy.

When NOT to use — and when it is actively misleading or dangerous

. - When the question is not yet specified. Running FFP against a vague aim produces a meaningless verdict; fitness is undefined without an estimand. Fix PICOTS first. - When the assessment is treated as a checkbox. A FFP memo that recites "relevant and reliable" without source-level evidence (provenance, code-list hit rates, missingness by site/time/arm, linkage denominators) is more dangerous than none, because it manufactures false confidence and is exactly the artifact a regulator will probe. The danger is laundering an unfit source through a process veneer. - When a fatal relevance gap is rationalized into a "mitigation." If the outcome is out-of-hospital cardiac death and the source has no death-index linkage, no sensitivity analysis rescues it — that is a no-go, not a go-with-mitigations. Mitigations are for measurable, bounded uncertainty (a quantitative-bias analysis for a validated-but-imperfect outcome algorithm), not for structurally absent data. - When reliability is assumed because the database is large or familiar. Size is not accuracy; a marquee claims database can still drop fee-for-service claims for Medicare Advantage enrollees, lag adjudication, reverse claims, or bundle services — all of which silently corrupt the very variables the study depends on.

Data-source operational depth

. - Administrative claims (FFS vs MA vs commercial): Relevance strengths are exposure (NDC + `fill_date` + `days_supply`) and healthcare utilization/cost; weaknesses are clinical severity, labs, vitals, and cause of death. Critical reliability failure modes: Medicare Advantage person-time lacks fee-for-service claims — encounter data are incomplete and inconsistently submitted, so an MA enrollee can look like a non-user or a non-utilizer purely from missingness; restrict to enrollees with the relevant benefit (A/B/D, or commercial medical+pharmacy) and exclude MA-only person-time unless complete encounter data are demonstrated. Other failure modes: adjudication lag and claim reversals (right-censor with a data-maturity buffer), bundled/capitated services that hide individual procedures, plan-switching that breaks continuous enrollment, and sample/mail-order fills that distort `days_supply`. Differential competing risks matter: in elderly claims, death competes with the outcome and may be captured only via the Medicare enrollment database, not the claim stream — verify the mortality source before trusting any time-to-event endpoint. - EHR: Relevance strengths are labs, vitals, problem lists, and clinician notes (severity, indication); the dominant reliability problem is encounter-driven, network-bounded capture — a patient who seeks care outside the system is differentially unobserved ("leakage"), so absence of a record is ambiguous (no event vs cared-for elsewhere). Structured fields are often sparse or entered inconsistently across sites; note availability varies by visit type. Linkage to claims is the standard fix for completeness, and explicit observation windows plus loss-to-follow-up handling are mandatory. - Registry: Relevance strength is adjudicated, clinically rich outcomes and disease staging (e.g., cancer registries); weakness is incomplete longitudinal pharmacy exposure and follow-up. Reliability turns on enrollment eligibility, case-ascertainment completeness, adjudication rules, and reporting lag. Almost always requires linkage to claims (for exposure/utilization) and to a death index (for mortality). - Linked claims–EHR–vital-records: The ideal substrate (EHR severity + claims completeness + reliable mortality) but linkage introduces selection (only the linkable subset, which may differ systematically) and date-reconciliation problems across order, fill, and service dates that must be resolved before time-zero assignment. Report the linkage denominator and compare linked vs unlinkable patients.

Worked example (claims-style logic)

Question (PICOTS fixed): among adults ≥18 with type-2 diabetes, does initiating a GLP-1 receptor agonist vs a DPP-4 inhibitor change 3-point MACE risk over 2 years? Candidate source: a commercial + Medicare fee-for-service claims database. (1) Relevance — population: confirm ≥2 T2D diagnoses are codeable and the age band is present. Exposure: both classes are identifiable by NDC with `fill_date` and `days_supply`, so new-user status (no prior fill in a 365-day washout) and on-treatment episodes are constructible. Outcome: 3-point MACE = nonfatal MI + nonfatal stroke + cardiovascular death; the MI/stroke components are claims-codeable, but CV death requires a death source — plain claims give an end-of-enrollment date, not a cause, so without National Death Index or Medicare-enrollment death linkage the outcome is only partially ascertainable: a relevance gap, not a reliability nuance. Confounders: HbA1c and BMI (key effect modifiers) are largely absent in claims — note this as a candidate for linkage or quantitative bias analysis. Follow-up: 2 years requires continuous A/B/D (or commercial) enrollment; check the median observable follow-up against the 2-year horizon. (2) Reliability: obtain ETL/provenance documentation and refresh date; right-censor with a 3-month maturity buffer for adjudication lag; exclude MA-only person-time because fee-for-service claims are missing there; profile missingness of `days_supply` and date fields by calendar quarter and by arm; verify the mortality source completeness against expected age-specific rates. (3) Verdict: go-with-mitigations — fit for nonfatal MACE components and exposure; conditionally fit for fatal MACE only if death-index linkage is secured; otherwise restrict the endpoint to nonfatal MACE or escalate to a linked source. (4) Pre-specified sensitivity analyses targeting the judgment-dependent thresholds: vary the washout (180 vs 365 days), the data-maturity buffer (1 vs 3 vs 6 months), the MACE algorithm definition (and apply a PPV-based quantitative bias analysis), and the MA-exclusion rule, reporting cohort counts and the estimate's stability at each step.

Worked example

Scenario

A research team wants to study whether a GLP-1 receptor agonist (a diabetes drug) reduces the risk of serious heart events compared with a DPP-4 inhibitor (another diabetes drug) over two years. The primary outcome is called 3-point MACE: nonfatal heart attack, nonfatal stroke, or cardiovascular death. Before writing a single line of analysis code, the team runs a fit-for-purpose assessment on a commercial plus Medicare fee-for-service claims database. The table below lists each assessment dimension, what the team checks, and whether it passes or raises a concern.

Dataset

Fit-for-purpose checklist for a GLP-1 vs DPP-4 MACE study in a commercial plus Medicare FFS claims database

Steps

• Work through the relevance checks first: confirm that the source can capture each piece of the PICOTS question before checking data quality.

• The nonfatal MACE components (heart attack, stroke) are identifiable by hospital diagnosis codes — these checks pass.

• Cardiovascular death is a fatal outcome and requires a cause-of-death source such as the National Death Index; plain claims only record when enrollment ended, not why the patient died — this is a relevance gap, not a quality nuance.

• HbA1c and BMI are clinical measurements rarely captured in claims, so the team notes them as a confounder gap requiring either a linked EHR or a sensitivity analysis.

• Move to reliability: Medicare Advantage enrollees in the database may appear to have no prescription fills simply because their insurer does not submit fee-for-service claims — including their person-time would silently corrupt the exposure measure, so it must be excluded.

• The maturity buffer check confirms that very recent claims are incomplete due to adjudication lag; censor the data three months before the extraction date.

• Summarize the verdict: the source is fit for the nonfatal MACE components and for exposure measurement; it is not fit for the full 3-point MACE endpoint unless death-index linkage is secured.

Result

Verdict: GO WITH MITIGATIONS for nonfatal MACE (heart attack + stroke); CONDITIONAL NO-GO for cardiovascular death until death-index linkage is confirmed. Key gap: fatal outcome ascertainment. Required mitigations before analysis: (1) exclude Medicare Advantage-only person-time, (2) apply a 3-month data-maturity censor, (3) secure death-index linkage or restrict the primary endpoint to nonfatal MACE. Pre-specified sensitivity analyses: vary the washout length (180 vs 365 days), vary the maturity buffer (1 vs 3 vs 6 months), test the MACE diagnosis algorithm with and without a PPV-based correction, and report cohort counts under each MA-exclusion rule.

Runnable example

import pandas as pd
import numpy as np

REQUIRED_FOLLOWUP_DAYS = 730   # 2-year estimand horizon
EXPOSURE_NDCS = {...}          # NDC set for the study + comparator drug classes
OUTCOME_DX = {...}             # validated MI/stroke code set for nonfatal MACE

def ffp_claims_profile(enroll, rx, dx, death):
    out = {}

    # --- RELIABILITY: MA-only person-time lacks fee-for-service claims -> must be excludable ---
    person_days = (enroll["enroll_end"] - enroll["enroll_start"]).dt.days.clip(lower=0)
    ma_only_days = person_days[enroll["plan_type"].eq("MA")].sum()
    out["pct_person_time_MA_only"] = 100 * ma_only_days / max(person_days.sum(), 1)
    out["pct_enrollees_with_AB_D_benefit"] = 100 * enroll.groupby("person_id")["ab_d"].max().mean()

    # --- RELEVANCE: follow-up duration vs the estimand horizon ---
    span = (enroll.groupby("person_id")
                  .apply(lambda g: (g["enroll_end"].max() - g["enroll_start"].min()).days))
    out["median_observable_followup_days"] = float(span.median())
    out["pct_with_full_horizon"] = 100 * (span >= REQUIRED_FOLLOWUP_DAYS).mean()

    # --- RELEVANCE: exposure capture (NDC + days_supply usable?) ---
    exp = rx[rx["ndc"].isin(EXPOSURE_NDCS)]
    out["pct_exposed_with_valid_days_supply"] = 100 * exp["days_supply"].between(1, 180).mean()
    out["n_exposed_persons"] = exp["person_id"].nunique()

    # --- RELEVANCE: nonfatal-outcome code-list hit rate ---
    out["n_persons_with_outcome_code"] = dx.loc[dx["dx_code"].isin(OUTCOME_DX), "person_id"].nunique()

    # --- RELEVANCE/RELIABILITY: fatal endpoint depends on a real death source w/ cause ---
    out["pct_deaths_with_known_source"] = 100 * death["death_source"].notna().mean()
    out["pct_deaths_with_cause"] = 100 * death["cause_of_death"].notna().mean()  # ~0 => CV death NOT ascertainable

    # --- RELIABILITY: date-field missingness by calendar quarter (data-maturity / lag signal) ---
    rx_q = rx.assign(q=rx["fill_date"].dt.to_period("Q"))
    out["fill_date_missing_by_quarter"] = (rx_q["fill_date"].isna()
                                           .groupby(rx_q["q"]).mean().mul(100).round(2).to_dict())
    return pd.Series(out)

# profile = ffp_claims_profile(enroll, rx, dx, death)
# Verdict logic: fatal MACE is fit ONLY if pct_deaths_with_cause is non-trivial; otherwise restrict to nonfatal MACE
# or escalate to a death-index-linked source. MA-only person-time must be excluded before any time-to-event analysis.

library(data.table)
REQUIRED_FOLLOWUP_DAYS <- 730L
EXPOSURE_NDCS <- c(...)   # study + comparator NDC set
OUTCOME_DX    <- c(...)   # validated nonfatal MACE code set

ffp_claims_profile <- function(enroll, rx, dx, death) {
  setDT(enroll); setDT(rx); setDT(dx); setDT(death)
  out <- list()

  # RELIABILITY: MA-only person-time lacks fee-for-service claims
  enroll[, pdays := pmax(as.integer(enroll_end - enroll_start), 0L)]
  out$pct_person_time_MA_only <- 100 * enroll[plan_type == "MA", sum(pdays)] / max(enroll[, sum(pdays)], 1L)
  out$pct_enrollees_with_AB_D <- 100 * mean(enroll[, .(any_abd = any(ab_d)), by = person_id]$any_abd)

  # RELEVANCE: observable follow-up vs the 2-year horizon
  span <- enroll[, .(d = as.integer(max(enroll_end) - min(enroll_start))), by = person_id]$d
  out$median_observable_followup_days <- median(span)
  out$pct_with_full_horizon <- 100 * mean(span >= REQUIRED_FOLLOWUP_DAYS)

  # RELEVANCE: exposure capture
  exp <- rx[ndc %in% EXPOSURE_NDCS]
  out$pct_exposed_valid_days_supply <- 100 * mean(exp$days_supply >= 1 & exp$days_supply <= 180)
  out$n_exposed_persons <- uniqueN(exp$person_id)

  # RELEVANCE: nonfatal-outcome hit rate
  out$n_persons_with_outcome_code <- uniqueN(dx[dx_code %in% OUTCOME_DX, person_id])

  # RELEVANCE/RELIABILITY: fatal endpoint depends on a death source carrying cause
  out$pct_deaths_with_known_source <- 100 * mean(!is.na(death$death_source))
  out$pct_deaths_with_cause <- 100 * mean(!is.na(death$cause_of_death))  # ~0 => CV death NOT ascertainable
  out
}
# Verdict: fatal MACE fit only if pct_deaths_with_cause is non-trivial; else restrict to nonfatal MACE or escalate
# to a death-index-linked source. Exclude MA-only person-time before any time-to-event analysis.