Fit-for-Purpose Data Assessment

Fit-for-purpose (FFP) data assessment

is the gatekeeping step that decides, before any analysis is programmed, whether a particular real-world data source can credibly answer a particular question. It is not a generic "data quality score" and it is not transferable: a database can be fit for purpose for a comparative drug-utilization study and entirely unfit for a comparative mortality study run on the same patients.

The assessment is organized around two axes made canonical by the FDA RWD guidance and operationalized by Gatto et al.'s Structured Process to Identify Fit-for-Purpose Data (SPIFD): relevance — does the source contain the population, exposure, outcome, key confounders, and follow-up duration the estimand demands? — and reliability — are those elements accurate, complete, traceable, and produced by a stable, documented data-curation (ETL) process?

The output is a documented go / no-go / go-with-mitigations verdict for one PICOTS-defined question, plus the specific sensitivity analyses that will probe the most judgment-dependent thresholds.

Core conceptual distinction

. FFP assessment sits upstream of, and is distinct from, three things it is often confused with. (1) vs database feasibility / attrition counting: a feasibility funnel tells you how many patients survive each eligibility step; FFP tells you whether the surviving cohort and its variables mean what the protocol needs them to mean. Feasibility is a necessary input to FFP, not a substitute.

(2) vs algorithm/outcome validation: validation estimates the operating characteristics (PPV, sensitivity) of one variable; FFP integrates those characteristics with relevance and curation evidence into a question-level decision. (3) vs a global data-quality grade: question-agnostic grading (completeness %, conformance checks) is a reliability input but cannot, by itself, declare fitness, because fitness is defined relative to a specific estimand. The decisive output of FFP is therefore not a number but a defensible decision tied to a question, with the residual risks named and mitigated.

Pros, cons, and trade-offs

(specific & comparative, naming the alternatives).

• vs proceeding straight to analysis on a convenient database: FFP forces relevance/reliability to be argued in protocol language before code is written, which is exactly what FDA and EMA reviewers expect and what prevents an expensive study from being rejected for an avoidable data limitation (e.g., MA-only person-time with no fee-for-service claims, no death linkage for a mortality endpoint). Cost: it is front-loaded work that delays the first results and requires data-provenance documentation the vendor may not readily supply. Prefer FFP for any regulatory-grade or HTA-facing study.
• vs a one-time, question-agnostic data-quality scorecard: a reusable scorecard is cheap and comparable across databases, but it systematically over- or under-states fitness because it ignores the estimand — a source with 99% completeness on labs is still unfit for a question that turns on outpatient mortality. FFP is question-specific and therefore more defensible, at the price of being non-transferable and needing re-execution for each new question. Prefer the scorecard only as a pre-screen to shortlist databases, then run FFP on the finalists.
• vs multi-database replication as the primary safeguard: running the analysis in several databases is powerful against database-specific artifacts, but it is reactive and expensive, and concordant-but-wrong results across sources that share a structural flaw (e.g., all lack reliable cause-of-death) give false reassurance. FFP is proactive and cheaper. Use both when stakes are high: FFP to qualify each source, replication to test robustness.

When NOT to use — and when it is actively misleading or dangerous

.

• When the question is not yet specified. Running FFP against a vague aim produces a meaningless verdict; fitness is undefined without an estimand. Fix PICOTS first.
• When the assessment is treated as a checkbox. A FFP memo that recites "relevant and reliable" without source-level evidence (provenance, code-list hit rates, missingness by site/time/arm, linkage denominators) is more dangerous than none, because it manufactures false confidence and is exactly the artifact a regulator will probe. The danger is laundering an unfit source through a process veneer.
• When a fatal relevance gap is rationalized into a "mitigation." If the outcome is out-of-hospital cardiac death and the source has no death-index linkage, no sensitivity analysis rescues it — that is a no-go, not a go-with-mitigations. Mitigations are for measurable, bounded uncertainty (a quantitative-bias analysis for a validated-but-imperfect outcome algorithm), not for structurally absent data.
• When reliability is assumed because the database is large or familiar. Size is not accuracy; a marquee claims database can still drop fee-for-service claims for Medicare Advantage enrollees, lag adjudication, reverse claims, or bundle services — all of which silently corrupt the very variables the study depends on.

Data-source operational depth

.

• Administrative claims (FFS vs MA vs commercial): Relevance strengths are exposure (NDC + `fill_date` + `days_supply`) and healthcare utilization/cost; weaknesses are clinical severity, labs, vitals, and cause of death. Critical reliability failure modes: Medicare Advantage person-time lacks fee-for-service claims — encounter data are incomplete and inconsistently submitted, so an MA enrollee can look like a non-user or a non-utilizer purely from missingness; restrict to enrollees with the relevant benefit (A/B/D, or commercial medical+pharmacy) and exclude MA-only person-time unless complete encounter data are demonstrated. Other failure modes: adjudication lag and claim reversals (right-censor with a data-maturity buffer), bundled/capitated services that hide individual procedures, plan-switching that breaks continuous enrollment, and sample/mail-order fills that distort `days_supply`. Differential competing risks matter: in elderly claims, death competes with the outcome and may be captured only via the Medicare enrollment database, not the claim stream — verify the mortality source before trusting any time-to-event endpoint.
• EHR: Relevance strengths are labs, vitals, problem lists, and clinician notes (severity, indication); the dominant reliability problem is encounter-driven, network-bounded capture — a patient who seeks care outside the system is differentially unobserved ("leakage"), so absence of a record is ambiguous (no event vs cared-for elsewhere). Structured fields are often sparse or entered inconsistently across sites; note availability varies by visit type. Linkage to claims is the standard fix for completeness, and explicit observation windows plus loss-to-follow-up handling are mandatory.
• Registry: Relevance strength is adjudicated, clinically rich outcomes and disease staging (e.g., cancer registries); weakness is incomplete longitudinal pharmacy exposure and follow-up. Reliability turns on enrollment eligibility, case-ascertainment completeness, adjudication rules, and reporting lag. Almost always requires linkage to claims (for exposure/utilization) and to a death index (for mortality).
• Linked claims–EHR–vital-records: The ideal substrate (EHR severity + claims completeness + reliable mortality) but linkage introduces selection (only the linkable subset, which may differ systematically) and date-reconciliation problems across order, fill, and service dates that must be resolved before time-zero assignment. Report the linkage denominator and compare linked vs unlinkable patients.

Worked example (claims-style logic)

Question (PICOTS fixed): among adults ≥18 with type-2 diabetes, does initiating a GLP-1 receptor agonist vs a DPP-4 inhibitor change 3-point MACE risk over 2 years? Candidate source: a commercial + Medicare fee-for-service claims database. (1) Relevance — population: confirm ≥2 T2D diagnoses are codeable and the age band is present. Exposure: both classes are identifiable by NDC with `fill_date` and `days_supply`, so new-user status (no prior fill in a 365-day washout) and on-treatment episodes are constructible.

Outcome

3-point MACE = nonfatal MI + nonfatal stroke + cardiovascular death; the MI/stroke components are claims-codeable, but CV death requires a death source — plain claims give an end-of-enrollment date, not a cause, so without National Death Index or Medicare-enrollment death linkage the outcome is only partially ascertainable: a relevance gap, not a reliability nuance. Confounders: HbA1c and BMI (key effect modifiers) are largely absent in claims — note this as a candidate for linkage or quantitative bias analysis.

Follow-up

2 years requires continuous A/B/D (or commercial) enrollment; check the median observable follow-up against the 2-year horizon. (2) Reliability: obtain ETL/provenance documentation and refresh date; right-censor with a 3-month maturity buffer for adjudication lag; exclude MA-only person-time because fee-for-service claims are missing there; profile missingness of `days_supply` and date fields by calendar quarter and by arm; verify the mortality source completeness against expected age-specific rates.

(3) Verdict: go-with-mitigations — fit for nonfatal MACE components and exposure; conditionally fit for fatal MACE only if death-index linkage is secured; otherwise restrict the endpoint to nonfatal MACE or escalate to a linked source. (4) Pre-specified sensitivity analyses targeting the judgment-dependent thresholds: vary the washout (180 vs 365 days), the data-maturity buffer (1 vs 3 vs 6 months), the MACE algorithm definition (and apply a PPV-based quantitative bias analysis), and the MA-exclusion rule, reporting cohort counts and the estimate's stability at each step.

Decision diagram

flowchart TD
  Q[PICOTS + estimand fixed<br/>one specific question] --> REL{Relevance:<br/>can the source capture<br/>population, exposure, outcome,<br/>confounders, follow-up?}
  REL -->|fatal gap, e.g. no death linkage<br/>for a mortality endpoint| NOGO[NO-GO<br/>wrong source for this question]
  REL -->|yes / partial| RELI{Reliability:<br/>accurate, complete, traceable?<br/>documented stable ETL?}
  RELI -->|exclude MA-only person-time,<br/>verify mortality source,<br/>profile missingness + lag| VERDICT{Fit-for-purpose verdict}
  VERDICT -->|all elements adequate| GO[GO]
  VERDICT -->|bounded, measurable uncertainty| MIT[GO WITH MITIGATIONS<br/>+ pre-specified sensitivity analyses]
  VERDICT -->|structural gap rationalized| NOGO
  MIT --> SENS[Sensitivity: washout length,<br/>data-maturity buffer, algorithm PPV/QBA,<br/>MA-exclusion rule]

flowchart LR
  subgraph Relevance[What the question needs]
    P[Population] --- E[Exposure] --- O[Outcome] --- C[Confounders] --- F[Follow-up]
  end
  Relevance --> Claims[Claims:<br/>+ exposure, utilization, cost<br/>- severity, labs, cause of death]
  Relevance --> EHR[EHR:<br/>+ labs, vitals, notes, severity<br/>- network leakage, sparse fields]
  Relevance --> Registry[Registry:<br/>+ adjudicated outcomes, staging<br/>- longitudinal exposure/follow-up]
  Claims --> Linked[Linked claims-EHR-death index:<br/>severity + completeness + mortality<br/>- linkage selection + date reconciliation]
  EHR --> Linked
  Registry --> Linked

Worked example

Scenario

A research team wants to study whether a GLP-1 receptor agonist (a diabetes drug) reduces the risk of serious heart events compared with a DPP-4 inhibitor (another diabetes drug) over two years. The primary outcome is called 3-point MACE: nonfatal heart attack, nonfatal stroke, or cardiovascular death. Before writing a single line of analysis code, the team runs a fit-for-purpose assessment on a commercial plus Medicare fee-for-service claims database. The table below lists each assessment dimension, what the team checks, and whether it passes or raises a concern.

Dataset

Fit-for-purpose checklist for a GLP-1 vs DPP-4 MACE study in a commercial plus Medicare FFS claims database

Steps

Result

Verdict: GO WITH MITIGATIONS for nonfatal MACE (heart attack + stroke); CONDITIONAL NO-GO for cardiovascular death until death-index linkage is confirmed. Key gap: fatal outcome ascertainment. Required mitigations before analysis:

1. exclude Medicare Advantage-only person-time,
2. apply a 3-month data-maturity censor,
3. secure death-index linkage or restrict the primary endpoint to nonfatal MACE. Pre-specified sensitivity analyses: vary the washout length (180 vs 365 days), vary the maturity buffer (1 vs 3 vs 6 months), test the MACE diagnosis algorithm with and without a PPV-based correction, and report cohort counts under each MA-exclusion rule.

Trade-offs

Runnable example

import pandas as pd
import numpy as np

REQUIRED_FOLLOWUP_DAYS = 730   # 2-year estimand horizon
EXPOSURE_NDCS = {...}          # NDC set for the study + comparator drug classes
OUTCOME_DX = {...}             # validated MI/stroke code set for nonfatal MACE

def ffp_claims_profile(enroll, rx, dx, death):
    out = {}

    # --- RELIABILITY: MA-only person-time lacks fee-for-service claims -> must be excludable ---
    person_days = (enroll["enroll_end"] - enroll["enroll_start"]).dt.days.clip(lower=0)
    ma_only_days = person_days[enroll["plan_type"].eq("MA")].sum()
    out["pct_person_time_MA_only"] = 100 * ma_only_days / max(person_days.sum(), 1)
    out["pct_enrollees_with_AB_D_benefit"] = 100 * enroll.groupby("person_id")["ab_d"].max().mean()

    # --- RELEVANCE: follow-up duration vs the estimand horizon ---
    span = (enroll.groupby("person_id")
                  .apply(lambda g: (g["enroll_end"].max() - g["enroll_start"].min()).days))
    out["median_observable_followup_days"] = float(span.median())
    out["pct_with_full_horizon"] = 100 * (span >= REQUIRED_FOLLOWUP_DAYS).mean()

    # --- RELEVANCE: exposure capture (NDC + days_supply usable?) ---
    exp = rx[rx["ndc"].isin(EXPOSURE_NDCS)]
    out["pct_exposed_with_valid_days_supply"] = 100 * exp["days_supply"].between(1, 180).mean()
    out["n_exposed_persons"] = exp["person_id"].nunique()

    # --- RELEVANCE: nonfatal-outcome code-list hit rate ---
    out["n_persons_with_outcome_code"] = dx.loc[dx["dx_code"].isin(OUTCOME_DX), "person_id"].nunique()

    # --- RELEVANCE/RELIABILITY: fatal endpoint depends on a real death source w/ cause ---
    out["pct_deaths_with_known_source"] = 100 * death["death_source"].notna().mean()
    out["pct_deaths_with_cause"] = 100 * death["cause_of_death"].notna().mean()  # ~0 => CV death NOT ascertainable

    # --- RELIABILITY: date-field missingness by calendar quarter (data-maturity / lag signal) ---
    rx_q = rx.assign(q=rx["fill_date"].dt.to_period("Q"))
    out["fill_date_missing_by_quarter"] = (rx_q["fill_date"].isna()
                                           .groupby(rx_q["q"]).mean().mul(100).round(2).to_dict())
    return pd.Series(out)

# profile = ffp_claims_profile(enroll, rx, dx, death)
# Verdict logic: fatal MACE is fit ONLY if pct_deaths_with_cause is non-trivial; otherwise restrict to nonfatal MACE
# or escalate to a death-index-linked source. MA-only person-time must be excluded before any time-to-event analysis.

library(data.table)
REQUIRED_FOLLOWUP_DAYS <- 730L
EXPOSURE_NDCS <- c(...)   # study + comparator NDC set
OUTCOME_DX    <- c(...)   # validated nonfatal MACE code set

ffp_claims_profile <- function(enroll, rx, dx, death) {
  setDT(enroll); setDT(rx); setDT(dx); setDT(death)
  out <- list()

  # RELIABILITY: MA-only person-time lacks fee-for-service claims
  enroll[, pdays := pmax(as.integer(enroll_end - enroll_start), 0L)]
  out$pct_person_time_MA_only <- 100 * enroll[plan_type == "MA", sum(pdays)] / max(enroll[, sum(pdays)], 1L)
  out$pct_enrollees_with_AB_D <- 100 * mean(enroll[, .(any_abd = any(ab_d)), by = person_id]$any_abd)

  # RELEVANCE: observable follow-up vs the 2-year horizon
  span <- enroll[, .(d = as.integer(max(enroll_end) - min(enroll_start))), by = person_id]$d
  out$median_observable_followup_days <- median(span)
  out$pct_with_full_horizon <- 100 * mean(span >= REQUIRED_FOLLOWUP_DAYS)

  # RELEVANCE: exposure capture
  exp <- rx[ndc %in% EXPOSURE_NDCS]
  out$pct_exposed_valid_days_supply <- 100 * mean(exp$days_supply >= 1 & exp$days_supply <= 180)
  out$n_exposed_persons <- uniqueN(exp$person_id)

  # RELEVANCE: nonfatal-outcome hit rate
  out$n_persons_with_outcome_code <- uniqueN(dx[dx_code %in% OUTCOME_DX, person_id])

  # RELEVANCE/RELIABILITY: fatal endpoint depends on a death source carrying cause
  out$pct_deaths_with_known_source <- 100 * mean(!is.na(death$death_source))
  out$pct_deaths_with_cause <- 100 * mean(!is.na(death$cause_of_death))  # ~0 => CV death NOT ascertainable
  out
}
# Verdict: fatal MACE fit only if pct_deaths_with_cause is non-trivial; else restrict to nonfatal MACE or escalate
# to a death-index-linked source. Exclude MA-only person-time before any time-to-event analysis.

%let horizon = 730;  /* 2-year estimand follow-up requirement */

/* RELIABILITY: share of person-time that is MA-only (lacks fee-for-service claims). */
proc sql;
  create table reliab_ma as
  select 100 * sum(case when plan_type='MA' then max(intck('day',enroll_start,enroll_end),0) else 0 end)
             / max(sum(max(intck('day',enroll_start,enroll_end),0)),1) as pct_person_time_MA_only
  from work.enroll;
quit;

/* RELEVANCE: observable follow-up per person vs the horizon. */
proc sql;
  create table fu as
  select person_id, intck('day', min(enroll_start), max(enroll_end)) as obs_days
  from work.enroll group by person_id;
quit;
proc sql;
  create table relev_fu as
  select median(obs_days) as median_obs_followup_days,
         100*mean(obs_days >= &horizon) as pct_with_full_horizon
  from fu;
quit;

/* RELEVANCE: exposure capture and nonfatal-outcome code hit counts. */
proc sql;
  create table relev_exp as
  select count(distinct r.person_id) as n_exposed_persons,
         100*mean(r.days_supply between 1 and 180) as pct_valid_days_supply
  from work.rx r inner join work.exp_ndc e on r.ndc = e.ndc;
  create table relev_out as
  select count(distinct d.person_id) as n_persons_with_outcome_code
  from work.dx d inner join work.out_dx o on d.dx_code = o.dx_code;
quit;

/* RELEVANCE/RELIABILITY: can a FATAL endpoint be ascertained? (cause-of-death presence) */
proc sql;
  create table relev_death as
  select 100*mean(death_source is not missing) as pct_deaths_with_source,
         100*mean(cause_of_death is not missing) as pct_deaths_with_cause /* ~0 => CV death NOT ascertainable */
  from work.death;
quit;

/* RELIABILITY: date-field missingness / data-maturity signal by calendar quarter. */
proc freq data=work.rx;
  format fill_date yyq6.;
  tables fill_date / missing nocum;  /* watch for thinning recent quarters = adjudication lag */
run;

Citations

1. [1]Gatto NM, Campbell UB, Rubinstein E, Jaksa A, Mattox P, Mo J, Reynolds RF. The Structured Process to Identify Fit-For-Purpose Data: A Data Feasibility Assessment Framework. Clinical Pharmacology & Therapeutics. 2022;111(1):122-134.
2. [2]Gatto NM, Vititoe SE, Rubinstein E, Reynolds RF, Campbell UB. A Structured Process to Identify Fit-for-Purpose Study Design and Data to Generate Valid and Transparent Real-World Evidence for Regulatory Uses. Clinical Pharmacology & Therapeutics. 2023;113(6):1235-1239.
3. [3]FDA. Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products. Guidance for Industry. 2024.

1. [4]Schneeweiss S, Patorno E. Conducting Real-world Evidence Studies on the Clinical Outcomes of Diabetes Treatments. Endocrine Reviews. 2021;42(5):658-690.

1. [5]Hernán MA, Robins JM. Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available. American Journal of Epidemiology. 2016;183(8):758-764.