Claims Outcome Algorithm PPV/Sensitivity Trade-off

In plain language

When researchers use insurance claims to count a medical event — say, heart attacks in a diabetes drug study — they have to write a rule that says which billing codes count as a real event. Every rule sits on a trade-off: a tight rule (requiring a specific primary-position hospital code plus a confirming procedure) catches mainly true events but misses some real ones; a loose rule (any mention of the code, anywhere) catches nearly all real events but also flags many false alarms. Positive predictive value (PPV) measures how often a flagged record is a true event, while sensitivity measures how often a true event actually got flagged. For studies comparing two treatments on a relative scale (like a risk ratio), researchers usually favor a high-PPV rule because false alarms dilute the comparison — but a high-PPV rule undercounts events, so absolute rates reported from it need a correction step using the validated PPV.

A claims-based outcome algorithm is the operational rule — diagnosis codes, code positions, encounter setting, procedure or lab confirmation, and timing windows — that converts raw administrative records into a binary "case / non-case" indicator for analysis. Because no code list perfectly captures a clinical event, every algorithm sits on a frontier defined by its positive predictive value (PPV) (of records flagged as cases, the fraction that are true cases), sensitivity (of true cases, the fraction the algorithm flags), and specificity (of true non-cases, the fraction correctly left unflagged). The named trade-off is the choice along that frontier: tightening the rule (e.g., requiring a primary-position inpatient diagnosis plus a confirmatory troponin or revascularization procedure for acute myocardial infarction) raises PPV but lowers sensitivity; loosening it (any-position diagnosis in any setting) raises sensitivity but admits false positives and lowers PPV.

Core conceptual distinction — why the trade-off is not symmetric, and why the estimand decides. Outcome misclassification distorts different estimands in different, predictable ways, and this is the single point a methodologist will check first. - Non-differential misclassification (the algorithm's PPV/sensitivity are the same in both exposure arms): for a relative effect (risk ratio, rate ratio, hazard ratio) with a binary, reasonably rare outcome, non-differential error biases the estimate toward the null in expectation — but it is not guaranteed monotone in small samples or with competing forms of error, so "it only attenuates" is a heuristic, not a license. Imperfect sensitivity alone (with PPV = 1, i.e., you miss cases but never invent them) leaves the rate ratio unbiased while biasing absolute rates downward; imperfect specificity (false positives) attenuates the ratio. This asymmetry is the practical reason a high-PPV, lower-sensitivity algorithm is often preferred for comparative (ratio) estimands: a small, exposure-balanced deficit in case capture costs little, whereas false positives directly dilute the contrast. - Differential misclassification (PPV or sensitivity differs by exposure) can bias a ratio estimate in either direction and is the dangerous case. It arises whenever the exposure changes the probability of being worked up or coded: a drug whose label prompts monitoring (e.g., LFTs, ECGs, imaging) generates more diagnostic encounters and thus more coded events in its arm — detection / surveillance bias — even if true incidence is identical. No amount of high-PPV tightening fixes differential capture; only a validation study that estimates the operating characteristics within each arm, a quantitative-bias correction, or a design fix (active comparator with similar monitoring) addresses it. - For absolute estimands (incidence rate, cumulative incidence, number needed to harm, decision-analytic event probabilities feeding a cost-effectiveness model), even non-differential error matters directly: a high-PPV/low-sensitivity algorithm systematically undercounts events and must be PPV/sensitivity-corrected before the number is used.

Pros, cons, and trade-offs (named alternatives)

- High-PPV narrow algorithm vs high-sensitivity broad algorithm. The narrow rule (primary-position, inpatient, procedure/lab-confirmed) yields clean contrasts for ratio estimands and is the default for safety signals where a false positive is costly; its cost is undercounted absolute risk and lost power when the true event is uncommon. The broad rule maximizes event capture (useful for screening, feasibility counts, or when sensitivity-corrected absolute rates are the target) but pays in lower PPV and, if capture is exposure-related, in differential bias. Prefer narrow for comparative ratio estimands and regulatory safety; prefer broad with a validation-based correction when an unbiased absolute rate is the deliverable. - vs adjudicated / chart-reviewed endpoints. Full medical-record adjudication (see `endpoint-adjudication-chart-review-rwe`) is the reference standard and removes the trade-off — but is infeasible at full-cohort scale, slow, and impossible in de-identified claims without linkage. The standard compromise is a claims algorithm validated against an adjudicated subsample to estimate PPV (and, with a sampling frame over non-flagged records, sensitivity), then either accept a high-PPV algorithm or quantitatively correct. - vs quantitative bias analysis on a fixed algorithm (`quantitative-bias-analysis-toolkit-rwe`, `misclassification-bias-correction-rwe`). Rather than chase a perfect code list, fix a transparent algorithm and correct the estimate using externally or internally estimated sensitivity/specificity/PPV (matrix correction, multiple imputation, or probabilistic bias analysis). Prefer correction when validation data or credible priors exist and the estimand is absolute or the misclassification may be differential.

When to use this trade-off explicitly (decision rules)

Whenever the outcome is derived from codes rather than adjudicated: pre-specify the algorithm and its expected PPV in the protocol/SAP, justify the position on the PPV–sensitivity frontier by the estimand (ratio → favor PPV; absolute → favor a known, correctable sensitivity), and pre-specify at least one sensitivity analysis that swaps a narrow for a broad definition (or vice versa). For regulatory safety endpoints, anchor to a published validation (PPV with CI) for the specific code set, setting, and population.

When NOT to use / when this is actively misleading

- Do not treat a PPV borrowed from a different population, coding era (ICD-9 vs ICD-10-CM), or care setting as your own. PPV depends on the prevalence of true cases among flagged records and is not transportable; a PPV measured in a referral hospital does not apply to a community FFS population. Specificity and sensitivity travel better but still drift across settings. - Do not claim a high-PPV algorithm "removes bias" when the threat is differential capture — it does not; you have a clean numerator in each arm but the arms were screened differently. - Do not apply a non-differential attenuation argument to an absolute rate, to a non-rare outcome, or to a case-finding window that interacts with follow-up time — the toward-the-null heuristic can fail. - Do not use any-position diagnosis codes from the index hospitalization as the outcome when the same codes also define eligibility or the exposure-triggering event — you will manufacture immortal-time-like or reverse-causation artifacts; require the event code to fall strictly after time zero.

Data-source operational depth (with real failure modes and workarounds)

- Claims (FFS vs Medicare Advantage vs commercial). Build the algorithm from diagnosis codes (with code position and place of service), procedure codes, and drug fills (`fill_date`, `days_supply`, NDC). Failure mode: in Medicare, MA-only person-time lacks complete fee-for-service claims, so events are silently under-ascertained for MA enrollees — differential if exposure correlates with plan type; workaround: restrict to FFS Parts A/B (and D for drug outcomes) or flag and sensitivity-analyze MA spans. Failure mode: same-day duplicate and reversed/denied claims inflate event counts; workaround: dedupe on `person_id`+`claim_date`+code and drop reversals before counting. Failure mode: adjudication lag and claims run-out truncate recent events; workaround: enforce a run-out buffer and a closed ascertainment window. Failure mode: differential competing risks — in elderly claims cohorts, the sicker arm dies before the non-fatal coded event can occur, so a naive cumulative-incidence comparison of the algorithm-defined outcome is biased; workaround: model with cause-specific or Fine–Gray methods (`competing-risks-cause-specific-fine-gray-rwe`). - EHR. Capture is encounter-driven: a true event managed entirely outside the system (an out-of-network ED visit) is invisible, lowering sensitivity differentially by who stays in-network. Structured fields (problem lists, labs) can confirm cases and raise PPV, but missing structured data and free-text-only documentation suppress sensitivity. Workaround: define an explicit observation window per patient and treat external-care leakage as informative; consider NLP for note-based confirmation and report capture by site and calendar time. - Registry. Often the strongest case definition (adjudicated, staged) but incomplete enrollment and eligibility rules limit denominator validity; outcome capture may stop at registry exit. Workaround: link to claims/EHR for continued follow-up and to a death index for fatal events and censoring. - Linked claims–EHR–vital records. The ideal substrate for measuring the algorithm's PPV/sensitivity (EHR/registry truth, claims breadth, reliable mortality), but linkage selection (only the linkable subset) and date discrepancies between service, claim, and adjudication dates must be reconciled before counting an event relative to time zero.

Worked claims example (Medicare AMI, with the actual PPV correction arithmetic)

Question: incidence of acute myocardial infarction (AMI) in a Medicare FFS cohort. (1) Continuous enrollment / washout: require Parts A and B FFS enrollment for 365 days before index; exclude MA-only person-time so the absence of a prior AMI is observed, not missing. Define incident AMI by requiring no AMI diagnosis in that 365-day baseline. (2) Algorithm (high-PPV variant): an AMI is a hospitalization with ICD-10-CM I21.x in the primary diagnosis position and length of stay ≥ 1 day (or in-hospital death), with the admission date strictly after time zero; this mirrors the definition Kiyota et al. validated against hospital-record review in Medicare, where the primary-position inpatient code achieved high PPV. (3) First-event coding: take the first qualifying admission per `person_id`; drop same-day duplicate and reversed claims. (4) Time windows: ascertain events only within the closed follow-up window (time zero to disenrollment, death, or a fixed end date minus a 90-day claims run-out buffer). (5) Validation + correction: in an adjudicated subsample, suppose the algorithm flags 100 records of which chart review confirms 90 — PPV = 0.90 (95% CI ≈ 0.82–0.95, Wilson). If the algorithm counts 1,000 AMIs in the cohort, the PPV-corrected true-case count is 1,000 × 0.90 = 900 events; propagating the PPV CI gives roughly 820–950 corrected events, which then feed the incidence rate (and any downstream cost or QALY model) instead of the raw 1,000. If sensitivity is also estimable from a sample of non-flagged records, the corrected count generalizes to corrected = observed × PPV ÷ sensitivity, and the gap between the high-PPV and a high-sensitivity broad definition becomes the headline sensitivity analysis.

Interpreting the output

From the worked example (simpler numbers): TP = 80, FP = 20, FN = 20, TN = 30. PPV = 80 / 100 = 0.80; sensitivity = 80 / 100 = 0.80. From the claims cohort, 200 flagged AMIs → PPV-corrected true-case count = 200 × 0.80 = 160. Correcting further for sensitivity: 160 / 0.80 ≈ 200 total true AMIs.

(1) Formal interpretation. PPV = 0.80 means that 20% of algorithm-flagged cases are false positives; they inflate the numerator of any incidence rate or event count. Under non-differential misclassification (PPV equal across arms), false positives dilute the contrast toward the null in a comparative relative- risk study — the PPV-corrected rate ratio will be further from 1.0 than the observed. Under differential misclassification (PPV higher in one arm), bias can go in either direction, and arm-specific PPV estimates are required. The PPV-corrected count (160) still understates total true AMIs because sensitivity is also < 1.0: missed true events (false negatives) reduce the denominator-adjusted count. The fully corrected count (≈ 200) requires both PPV and sensitivity to be estimable from the validation.

(2) Practical interpretation. Reporting raw algorithm-flagged counts as true event counts overstates AMI incidence by ≈ 25% in this example (200 flagged vs 160 confirmed). For a decision-analytic model or a cost-per-event calculation, the PPV-corrected figure is the appropriate input. For a comparative HR, the direction of bias matters more than the absolute count — analysts must state and justify the differential vs non-differential assumption before interpreting any corrected ratio.

Worked example

Scenario

A Medicare claims study is counting acute heart attacks (AMIs) in a cohort of older adults with Type 2 diabetes. The research team uses a high-PPV algorithm: a primary-position inpatient ICD-10-CM code starting with I21 (the AMI code family). The algorithm flags 200 hospitalizations across the full cohort. To validate it, the team pulls the actual hospital charts for 100 of those flagged records and has a cardiologist confirm whether a true AMI occurred. The cardiologist also reviews 50 records the algorithm did NOT flag, and finds 20 true AMIs the algorithm missed. The 2x2 table below shows the result. The goal is to compute PPV and sensitivity, then explain what the trade-off means for this study.

Dataset

2x2 validation table: algorithm result vs. chart-review truth for the 150 reviewed records (100 flagged + 50 unflagged).

Steps

• Identify the four cells: TP (true positives) = 80, the records the algorithm flagged AND the chart confirms as true AMIs. FP (false positives) = 20, flagged by the algorithm but NOT confirmed as AMIs. FN (false negatives) = 20, NOT flagged but the chart shows a real AMI occurred. TN (true negatives) = 30, not flagged and confirmed not to be AMIs.

• Compute PPV: divide TP by all flagged records. PPV = TP / (TP + FP) = 80 / (80 + 20) = 80 / 100 = 0.80. This means 80 out of every 100 records the algorithm flags are genuine heart attacks.

• Compute sensitivity: divide TP by all true AMIs found in the reviewed subsample. Sensitivity = TP / (TP + FN) = 80 / (80 + 20) = 80 / 100 = 0.80. This means the algorithm captures 80 out of every 100 real AMIs in the reviewed records.

• In this particular example both values happen to equal 0.80, but they measure different things and will usually differ. Changing the algorithm rule changes them in opposite directions: tightening the rule (e.g., also requiring a troponin procedure code) would push PPV higher (fewer false alarms) but push sensitivity lower (more missed cases).

• Apply the PPV-vs-sensitivity tradeoff: the study reports a relative effect (a hazard ratio comparing two drugs). For that purpose, the team accepts the 0.80 PPV rule because false alarms dilute the comparison roughly equally in both arms. However, if the team also wants to report the absolute rate of AMIs per 1,000 patient-years, the raw algorithm count is too low — sensitivity of 0.80 means 20 percent of real AMIs were missed. To correct the absolute rate, the team would divide the PPV-corrected count by sensitivity: corrected true events = (observed events x PPV) / sensitivity.

Result

PPV = 80 / (80 + 20) = 80 / 100 = 0.80. Sensitivity = 80 / (80 + 20) = 80 / 100 = 0.80. If the full-cohort algorithm flagged 200 AMIs, the PPV-corrected true-case estimate is 200 x 0.80 = 160 true AMIs. Because sensitivity is also 0.80, the algorithm missed roughly 20 percent of real events, so the PPV-corrected count still understates total true AMIs — for an absolute rate the corrected estimate would be 160 / 0.80 = 200 total true AMIs estimated in the cohort.

Runnable example

import pandas as pd
import numpy as np
from scipy.stats import norm

AMI_CODES = ("I21",)  # ICD-10-CM AMI family; match on prefix

def derive_outcome(dx: pd.DataFrame, cohort: pd.DataFrame,
                   primary_only: bool = True, settings=("IP",)) -> pd.DataFrame:
    """First incident outcome per person, strictly after time zero, within the ascertainment window.

    primary_only=True, settings=('IP',)        -> high-PPV narrow algorithm
    primary_only=False, settings=('IP','OP','ED') -> high-sensitivity broad algorithm
    """
    ev = dx[dx["dx_code"].str.startswith(AMI_CODES)].copy()
    if primary_only:
        ev = ev[ev["dx_position"] == 1]
    ev = ev[ev["place_of_service"].isin(settings)]

    ev = ev.merge(cohort[["person_id", "index_date", "fu_end"]], on="person_id", how="inner")
    # Event must fall strictly after time zero and within the closed follow-up window.
    ev = ev[(ev["claim_date"] > ev["index_date"]) & (ev["claim_date"] <= ev["fu_end"])]
    # First qualifying event per person (incident).
    first = (ev.sort_values(["person_id", "claim_date"])
               .groupby("person_id", as_index=False)
               .first()[["person_id", "claim_date"]]
               .rename(columns={"claim_date": "event_date"}))
    return first

def wilson_ci(x: int, n: int, alpha: float = 0.05):
    """Wilson score interval for a proportion (robust for PPV near 0/1 and small validation samples)."""
    if n == 0:
        return (np.nan, np.nan)
    z = norm.ppf(1 - alpha / 2)
    p = x / n
    denom = 1 + z**2 / n
    center = (p + z**2 / (2 * n)) / denom
    half = (z * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2))) / denom
    return (center - half, center + half)

def ppv_corrected_count(validated: pd.DataFrame, observed_events: int):
    """Estimate PPV from the adjudicated subsample and apply it to the full-cohort flagged count.

    Returns the corrected true-case count with an interval propagated from the PPV Wilson CI.
    If a sensitivity estimate is available, divide the corrected count by sensitivity to recover
    true incidence: corrected_true = observed * PPV / sensitivity.
    """
    flagged = validated[validated["flagged"] == 1]
    tp = int(flagged["true_case"].sum())
    n_flagged = int(len(flagged))
    ppv = tp / n_flagged if n_flagged else np.nan
    lo, hi = wilson_ci(tp, n_flagged)
    return {
        "ppv": ppv, "ppv_ci": (lo, hi),
        "corrected_events": observed_events * ppv,
        "corrected_events_ci": (observed_events * lo, observed_events * hi),
    }

library(data.table)

AMI_RE <- "^I21"  # ICD-10-CM AMI family

derive_outcome <- function(dx, cohort, primary_only = TRUE, settings = c("IP")) {
  setDT(dx); setDT(cohort)
  ev <- dx[grepl(AMI_RE, dx_code)]
  if (primary_only) ev <- ev[dx_position == 1L]
  ev <- ev[place_of_service %chin% settings]
  ev <- merge(ev, cohort[, .(person_id, index_date, fu_end)], by = "person_id")
  # Strictly after time zero and inside the closed ascertainment window.
  ev <- ev[claim_date > index_date & claim_date <= fu_end]
  setorder(ev, person_id, claim_date)
  ev[, .(event_date = claim_date[1L]), by = person_id]  # first incident event per person
}

wilson_ci <- function(x, n, alpha = 0.05) {
  if (n == 0) return(c(NA_real_, NA_real_))
  z <- qnorm(1 - alpha / 2); p <- x / n; denom <- 1 + z^2 / n
  center <- (p + z^2 / (2 * n)) / denom
  half <- (z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2))) / denom
  c(center - half, center + half)
}

ppv_corrected_count <- function(validated, observed_events) {
  setDT(validated)
  flagged <- validated[flagged == 1L]
  tp <- sum(flagged$true_case); n_flagged <- nrow(flagged)
  ppv <- if (n_flagged > 0) tp / n_flagged else NA_real_
  ci <- wilson_ci(tp, n_flagged)
  list(ppv = ppv, ppv_ci = ci,
       corrected_events = observed_events * ppv,
       corrected_events_ci = observed_events * ci)
}