Outcome Algorithm Construction

In plain language

An outcome algorithm is a precise written rule that tells a computer how to search a claims database and decide which patients had a specific medical event — for example, a heart attack. Because the database never saw the patient in person, the rule uses billing codes, the type of hospital visit, and where the code appears on the claim to make that determination. The algorithm can be wrong in both directions (flagging patients who never had the event, or missing patients who did), so researchers check its accuracy against actual medical records and report that check as part of the study.

Outcome algorithm construction

is the disciplined translation of a protocol-defined endpoint (e.g., "acute myocardial infarction," "incident heart failure hospitalization," "all-cause mortality") into an executable, fully specified case-finding rule over routinely collected data, plus the validation study that tells you how often that rule is right. An algorithm is a tuple of decisions: which code systems and code lists (ICD-10-CM, CPT/HCPCS, NDC, LOINC, SNOMED), which care settings (inpatient principal vs any-position, emergency, outpatient), how many encounters and in what time window (the classic "1 inpatient OR 2 outpatient ≥7 days apart"), whether labs/vitals/meds are required to confirm, and how repeated codes for the same true event are de-duplicated into one incident outcome. The algorithm is not credible until you attach its operating characteristics — positive predictive value (PPV), sensitivity, and specificity — measured against chart review, registry adjudication, or a death index, because every downstream rate, hazard ratio, and ICER inherits the algorithm's measurement error.

Core conceptual distinction

Three quantities are distinct and routinely confused. (1) PPV — among algorithm-flagged patients, the fraction who truly have the outcome — governs whether your numerator is contaminated by false positives. (2) Sensitivity — among true cases, the fraction the algorithm catches — governs how many events you miss. (3) Specificity — among true non-cases, the fraction correctly excluded — drives false positives in the much larger non-diseased denominator and is the dominant driver of bias for rare outcomes. The critical, often-missed point: non-differential outcome misclassification does not reliably bias toward the null for ratio measures when specificity is imperfect and the outcome is rare — a tiny drop in specificity floods a rare-event numerator with false positives from the huge at-risk pool, and if that misclassification is differential by exposure (e.g., treated patients are surveilled more and thus coded more), the bias can go in any direction and magnitude. Construction therefore has two estimands behind it: the measurement estimand (PPV/sensitivity/specificity of the algorithm) and the substantive estimand (the rate or effect on the true outcome), connected by quantitative bias analysis.

Pros, cons, and trade-offs

(specific and comparative). - High-PPV ("specific") algorithms vs high-sensitivity ("sensitive") algorithms: A strict rule (e.g., inpatient principal diagnosis + confirmatory troponin) maximizes PPV, minimizes false positives, and is the right default for effect estimation, because a clean numerator protects ratio measures. Cost: it sacrifices sensitivity and undercounts incidence, so it is wrong for burden-of-illness / incidence reporting. A broad rule (any-position diagnosis in any setting) maximizes sensitivity for case-finding and surveillance but admits rule-out and historical codes. Prefer high-PPV for comparative effectiveness/safety HRs; prefer high-sensitivity for screening, signal detection, or when you will bias-correct with known sensitivity/specificity. - vs naive use of a single diagnosis code: A validated multi-criterion algorithm dramatically reduces misclassification versus "any ICD code = case," at the cost of code complexity and the need for a validation substudy. Always prefer the validated algorithm; an unvalidated code count is indefensible to FDA/EMA. - vs end-to-end chart adjudication of every potential event: Adjudication (see `endpoint-adjudication-chart-review-rwe`) is the gold standard but is expensive and infeasible at scale; algorithm construction + a sampled validation substudy buys most of the accuracy at a fraction of the cost, and the PPV from the substudy lets you bias-correct the full cohort.

When to use

(decision rules). Any RWE study whose endpoint is derived from claims or EHR structured data: build the algorithm from pre-specified code lists, lock it before looking at outcome-by-exposure associations, and validate a random sample against a gold standard (or import published operating characteristics for the same algorithm, setting, and era). Report the algorithm and its PPV/sensitivity in the protocol and the manuscript per RECORD/Sentinel norms.

When NOT to use — and when it is actively misleading or dangerous

(decision rules). - Borrowing validation metrics across a different data source, code era, or population. A PPV established in Medicare FFS inpatient data does not transport to a commercial outpatient setting or across the ICD-9→ICD-10 transition; a borrowed PPV applied to the wrong base rate gives a falsely precise, biased correction. - Imperfect specificity with a rare outcome and no bias correction. Reporting a crude incidence or HR from a sensitive algorithm without specificity-based correction can be badly biased; this is the dangerous case where "more cases" silently means "more false positives." - Differential surveillance/coding by exposure. When one arm is monitored or coded more intensively (new drug under a REMS, sicker comparator seen more often), outcome ascertainment is differential and no simple non-differential correction is valid — switch to blinded adjudication or a negative-control outcome to detect it. - Prevalent codes masquerading as incident events. Without a clean baseline washout, chronic-condition codes (heart failure, cancer history "Z" codes) recur every visit and an "incident outcome" algorithm will count follow-up of an old diagnosis as a new event — manufacturing immortal-time-like and ascertainment artifacts.

Data-source operational depth

(claims vs EHR vs registry vs linked). - Claims (FFS vs MA): The unit is the claim line with diagnosis position, place-of-service, and revenue code. Inpatient principal diagnoses are far more specific than any-position or outpatient codes (which include rule-out and historical coding). Require continuous medical enrollment so absence of a code is a true negative, not unobserved care. Failure mode: Medicare Advantage encounter data are incomplete and inconsistently submitted relative to fee-for-service claims, so person-time under MA can fabricate false negatives (missed events) and distort sensitivity — restrict to FFS person-time or model the differential capture. Failure mode: differential competing risks — in elderly claims cohorts, death removes people from the at-risk set before the outcome can be coded, and if death rates differ by exposure the algorithm's observed incidence is distorted; pair the construction with a death index and a competing-risk framing. Failure mode: immortal time in procedure-anchored algorithms — defining the outcome relative to a procedure that itself requires survival builds in guaranteed event-free time. - EHR: Richer (labs, vitals, notes) so you can require confirmatory features (troponin for MI, ejection fraction for HF), but capture is visit-driven and fragmented across systems; a patient who gets the event at an out-of-network hospital is a false negative. Define the observation window explicitly and treat loss to follow-up as informative. NLP-derived phenotypes (see `ehr-phenotyping-algorithms-rwe`) raise sensitivity but need their own validation. - Registry: Outcomes are often adjudicated (cancer stage, MI by universal definition) — the strongest source for the truth — but pharmacy/exposure and out-of-registry events are weak; link to claims for completeness and to a death index for mortality. - Linked claims–EHR–vital records: The ideal substrate (EHR confirmatory features + claims completeness + reliable death) but linkage selects the linkable subset and introduces date discrepancies among service, fill, and death dates that must be reconciled before the first-event date is set.

Worked claims example (PPV-adjusted incidence and bias correction)

Endpoint: incident hospitalized acute myocardial infarction (AMI) among new initiators of a study drug in a Medicare FFS + commercial database. (1) Code list and rule: AMI = an inpatient claim with ICD-10-CM I21.x in the principal position; this high-PPV rule is chosen because the estimand is a comparative hazard ratio. (2) Continuous enrollment: require medical enrollment from index through the event so absence of a claim is a true negative; exclude MA-only person-time because MA encounter capture is incomplete. (3) Incident-event de-duplication: require a 365-day clean baseline with no I21.x/I22.x and collapse same-admission transfers and re-codes within a 30-day window into one event (see `acute-event-deduplication-window-rwe` and `hospitalization-transfer-collapse-rwe`) so a single infarction with a hospital transfer is not double-counted. (4) First-event date: admission date of the first qualifying hospitalization; censor at disenrollment, death, end of data. (5) Validation substudy: pull charts for a random sample of n=200 algorithm-flagged admissions; suppose 174 are true AMIs, giving PPV = 174/200 = 0.87 (95% CI ≈ 0.82–0.91 by exact binomial). (6) Bias-corrected count: if the algorithm flags 1,000 AMIs over 50,000 person-years, the estimated true count is 1,000 × 0.87 = 870 true AMIs, so the PPV-corrected incidence is 870 / 50,000 = 17.4 per 1,000 person-years versus the naive 20.0 per 1,000 — a 13% overstatement removed. (7) Sensitivity correction: if an independent chart-review of true AMIs yields sensitivity = 0.80, the algorithm misses ~20% of events; under non-differential ascertainment the exposed/unexposed PPV- and sensitivity-corrected counts feed a corrected rate ratio, and the gap between the naive and corrected ratio is itself the quantitative-bias-analysis result that goes in the sensitivity-analysis table (see `misclassification-bias-correction-rwe`).

Interpreting the output

. The AMI algorithm flags 1,000 candidate events from 50,000 person-years; chart review of 200 sampled positives confirms 174 true AMIs, giving PPV = 0.87. Naive incidence is 20.0 per 1,000 person-years; after PPV correction — 1,000 × 0.87 = 870 estimated true AMIs — the corrected incidence is 17.4 per 1,000 person-years, removing a 13% overstatement.

Formal interpretation: the algorithm's output is not an AMI count — it is an AMI candidate count with a measured false-positive rate. PPV = 0.87 means 13 of every 100 flagged events are false positives; ignoring this inflates the numerator and biases any rate or ratio toward the null under non-differential misclassification. The tradeoff is intentional: a strict (high-PPV) rule routes well for ratio estimands such as a hazard ratio where false positives in both arms roughly cancel; a broad (high-sensitivity) rule routes better for incidence burden estimation where missing events matters more than false-positive contamination. Choose the operating point before unblinding outcomes, pre-specify it in the SAP, and present both the naive and PPV-corrected rates as the primary table.

Practical interpretation: the 2.6-per-1,000-PY gap between naive and corrected rates is a tangible audit trail — if a safety signal appears in the naive analysis and shrinks substantially after correction, reviewers know the algorithm's precision is load-bearing. Always propagate the uncertainty in PPV (its binomial confidence interval) through the correction via a Monte Carlo draw or analytic error formula.

Worked example

Scenario

You are building a rule to find patients who had an acute heart attack (acute MI) in a claims database, so you can count how often this event happens. Your rule says: a patient counts as having a heart attack only if they have an inpatient hospital claim where the heart-attack code (ICD-10-CM I21) is in the principal diagnosis position. The table below shows five candidate patients. After the algorithm runs, you check a sample of flagged patients against their real medical records to see how often the rule was correct.

Dataset

Five candidate patients from the claims diagnosis table. Each row is one claim line.

Steps

• The rule requires THREE things to all be true at once: care_setting = inpatient, dx_position = principal, and the code starts with I21.

• Patient 1001 passes all three checks — inpatient stay, principal position, I21.0 — so the algorithm flags them as a heart attack.

• Patient 1002 fails the setting check: the claim is outpatient, so a code appearing there may just mean the doctor was ruling out a heart attack, not confirming one. Not flagged.

• Patient 1003 fails the position check: the I21.0 code is in a secondary slot, meaning it was a side condition, not the primary reason for admission. Not flagged.

• Patient 1004 fails the code check: I25.10 is chronic coronary artery disease — a different condition entirely. Not flagged.

• Patient 1005 passes all three checks — inpatient, principal, I21.1 — so they are also flagged.

• The algorithm flags 2 patients (1001 and 1005). A researcher then pulls the medical records for both. Suppose both records confirm a real heart attack. That gives PPV = 2 true cases / 2 flagged = 1.00 for this tiny sample. In a real study with hundreds of flagged patients, PPV is typically around 0.85–0.90 for this kind of strict rule, meaning roughly 1 in 8 flagged patients did not actually have a heart attack.

Result

The algorithm flags 2 of 5 candidates (patients 1001 and 1005) by requiring inpatient principal-position I21 codes. The three patients who were not flagged were excluded for the correct reasons — wrong setting, wrong position, or wrong code. When a researcher checks real medical records for a random sample of flagged patients (the PPV validation step), they can calculate what fraction were true heart attacks and use that number to adjust the final event count.

Runnable example

import pandas as pd
import numpy as np

AMI_CODES = ("I21", "I22")        # locked ICD-10-CM prefixes for acute MI
WASHOUT_DAYS = 365                # clean baseline that makes the event incident
DEDUP_DAYS = 30                   # collapse same-event re-codes / transfers into one event

def build_outcome(dx, enroll, index):
    # High-PPV rule: inpatient PRINCIPAL-position AMI code.
    flagged = dx[(dx["care_setting"] == "inpatient") &
                 (dx["dx_position"] == "principal") &
                 (dx["icd10"].str.startswith(AMI_CODES))].copy()
    flagged = flagged.merge(index, on="person_id")

    # Incident: drop people with the same code in the WASHOUT before index; keep events after index only.
    prior = flagged[flagged["claim_date"] < flagged["index_date"]]
    prevalent = prior.loc[prior["claim_date"] >= prior["index_date"] -
                          pd.Timedelta(days=WASHOUT_DAYS), "person_id"].unique()
    post = flagged[(flagged["claim_date"] >= flagged["index_date"]) &
                   (~flagged["person_id"].isin(prevalent))].sort_values(["person_id", "claim_date"])

    # First qualifying admission = the incident event date; collapse re-codes within DEDUP_DAYS.
    first = post.groupby("person_id", as_index=False).first().rename(columns={"claim_date": "event_date"})

    # Observable, FFS-only follow-up: censor MA-only spans so a missing code is a true negative.
    e = enroll[~enroll["ma_only"]].merge(index, on="person_id")
    e["pt_days"] = (np.minimum(e["enroll_end"], e["index_date"] + pd.Timedelta(days=365)) -
                    np.maximum(e["enroll_start"], e["index_date"])).dt.days.clip(lower=0)
    person_years = e.groupby("person_id")["pt_days"].sum().sum() / 365.25
    return first[["person_id", "event_date"]], person_years

def ppv_corrected_incidence(events, person_years, chart):
    ppv = chart["true_case"].mean()                       # PPV among flagged-and-charted events
    n_obs = len(events)
    n_true = n_obs * ppv                                  # bias-corrected true count = observed x PPV
    return {"ppv": ppv,
            "naive_rate_per_1000_py": 1000 * n_obs / person_years,
            "corrected_rate_per_1000_py": 1000 * n_true / person_years}

library(data.table)
AMI_CODES   <- "^I2[12]"   # locked ICD-10-CM regex for acute MI
WASHOUT     <- 365L
DEDUP_DAYS  <- 30L

build_outcome <- function(dx, enroll, index) {
  setDT(dx); setDT(enroll); setDT(index)
  flagged <- dx[care_setting == "inpatient" & dx_position == "principal" &
                grepl(AMI_CODES, icd10)]
  flagged <- merge(flagged, index, by = "person_id")

  # Incident: exclude prevalent (same code in washout before index); keep post-index events.
  prevalent <- flagged[claim_date < index_date & claim_date >= index_date - WASHOUT,
                       unique(person_id)]
  post <- flagged[claim_date >= index_date & !(person_id %chin% prevalent)]
  setorder(post, person_id, claim_date)
  first <- post[, .(event_date = claim_date[1L]), by = person_id]   # first admission, dedup by taking earliest

  # FFS-only observable follow-up so a missing code is a true negative.
  e <- merge(enroll[ma_only == FALSE], index, by = "person_id")
  e[, pt_days := as.numeric(pmin(enroll_end, index_date + 365) -
                            pmax(enroll_start, index_date))]
  e[pt_days < 0, pt_days := 0]
  person_years <- sum(e$pt_days) / 365.25
  list(events = first, person_years = person_years)
}

ppv_corrected_incidence <- function(events, person_years, chart) {
  ppv    <- mean(chart$true_case)                  # PPV among flagged-and-charted events
  n_obs  <- nrow(events)
  n_true <- n_obs * ppv                            # corrected true count = observed x PPV
  list(ppv = ppv,
       naive_rate_per_1000_py     = 1000 * n_obs  / person_years,
       corrected_rate_per_1000_py = 1000 * n_true / person_years)
}