Algorithm Validation

In plain language

An algorithm validation study checks whether a rule written in claims or electronic health record data actually identifies the right patients. To find cases of a disease in insurance data, researchers write a rule — for example, "two or more diagnosis codes for Type 2 diabetes within any 12-month period" — but that rule will sometimes flag patients who don't truly have the disease and miss patients who do. A validation study solves this by pulling a random sample of flagged and un-flagged records, having clinicians review the actual charts to confirm the true diagnosis, and then measuring how often the rule agreed with the chart. The result is a report card — numbers like sensitivity, specificity, and positive predictive value — that tells every future researcher exactly how trustworthy the rule is before they use it to study treatment effects or costs.

Algorithm validation

quantifies how well a computable definition applied to routinely-collected data (an ICD/CPT/NDC/lab rule for a diagnosis, exposure, or outcome) reproduces the truth as captured by a reference standard (chart review, registry linkage, adjudication committee, or a richer linked source). The deliverable is a set of operating characteristics from a 2x2 table of algorithm flag against adjudicated truth — positive predictive value (PPV), sensitivity (Se), specificity (Sp), and negative predictive value (NPV) — each with a binomial confidence interval. Validation is not a checkbox: it is the input to quantitative bias analysis (QBA) that propagates measurement error into the effect estimate. An algorithm is "validated" only relative to a population, an era, and a use; the same code list can be excellent for case identification in one setting and dangerously biased in another.

Core conceptual distinction — which metric you can estimate, and which you can transport

The metric you can estimate depends on your sampling frame. If you sample only algorithm-positive charts (the common, cheap "PPV study"), you can estimate PPV but not Se or Sp, because you never observe truth among algorithm-negatives. To estimate Se and Sp you must also sample (a usually weighted subset of) algorithm-negatives and adjudicate them — far more expensive and the reason most published validations report PPV only. The critical, frequently-missed asymmetry: Se and Sp are properties of the algorithm-versus-truth classification and are approximately portable across populations; PPV and NPV are not. By Bayes' rule, PPV = Sepi / (Sepi + (1 - Sp)(1 - pi)), where pi is the true prevalence in the target* population. A PPV of 0.90 measured in a high-prevalence inpatient validation sample can collapse to 0.50 in a low-prevalence ambulatory cohort even though the algorithm is unchanged. This is why correction methods that travel (e.g., the Se/Sp matrix correction) are built on Se/Sp, and why porting a PPV number across data sources is a recurring error.

The estimand it feeds, and why misclassification direction matters

Validation exists to defend a downstream causal or descriptive estimand — an incidence rate, a risk ratio, a hazard ratio. Non-differential outcome misclassification (Se/Sp identical across exposure arms) with imperfect specificity biases a risk ratio toward the null in expectation, so a "null" finding from an unvalidated, low-specificity outcome is uninterpretable. Differential misclassification (Se/Sp differing by exposure — e.g., treated patients are surveilled more and so their outcomes are coded more completely) biases in an unpredictable direction and magnitude, and a correction that assumes non-differentiality can move the estimate further from the truth. The whole point of validation-plus-QBA is to replace the comforting but false "non-differential bias is conservative" reflex with an explicit, quantified adjustment.

Pros, cons, and trade-offs

- vs face-validity / citing someone else's number: A study-specific internal validation removes the heroic transport assumption and yields Se/Sp/PPV calibrated to your data, era, and population. Cost: charts, adjudicators, and IRB-permitted re-identification. Prefer internal validation for primary endpoints in regulatory or HTA submissions; borrowing an external estimate is acceptable only as a sensitivity-analysis anchor, never as the sole basis, and only when the source population's prevalence and coding practices are demonstrably similar. - vs a single high-PPV code list (no Se/Sp): A PPV-only design is cheap and answers "of those I flagged, how many are real?" — sufficient when you only need confirmed cases (case-finding) and accept lost power. It is inadequate when you need to correct an effect estimate, because the matrix and most probabilistic corrections require Se and Sp. Prefer a Se/Sp design whenever the validated variable is the outcome or exposure in a comparative analysis. - vs probabilistic bias analysis (PBA): A simple matrix (deterministic) correction plugs point estimates of Se/Sp into a back-calculation and is transparent and fast, but understates uncertainty and can produce impossible (negative) cell counts when Se/Sp estimates are imprecise. PBA draws Se/Sp from their sampling distributions (e.g., Beta priors from the validation 2x2) and yields a simulation interval that honestly combines random and systematic error. Prefer PBA when the validation sample is small or the correction must appear in a defensible inference; use the matrix correction for a quick directional check. - vs latent-class / Bayesian no-gold-standard models: When the "reference standard" is itself imperfect (chart review misses, adjudication disagrees), assuming it is truth biases Se/Sp. Latent-class models estimate algorithm and reference-standard accuracy jointly without a perfect gold standard, at the cost of strong conditional-independence assumptions that are easy to violate (two ICD-based references that share the same coding error are not independent).

When to use

Any analysis where a study variable is defined by a data-derived algorithm and measurement error could change the conclusion: the primary outcome of a comparative safety/effectiveness study, the exposure phenotype, a subgroup-defining comorbidity, or a regulator/HTA-facing endpoint. Validate the specific algorithm you will run (operators, time windows, code versions), in a sample drawn from the same source and era, with a reference standard appropriate to the construct.

When NOT to use / when it is actively misleading or dangerous

- Transporting PPV across populations or eras. Moving a PPV from a hospitalized validation sample to an ambulatory cohort, or from ICD-9 to ICD-10 coding, or from a commercial to a Medicare population, ignores the prevalence and coding-practice dependence above. If you must reuse external accuracy, transport Se/Sp (more stable) and re-derive PPV at the target prevalence — never reuse PPV directly. - Non-representative validation sample (selection on the algorithm or on adjudicability). If chartable patients differ systematically from the analytic cohort (e.g., only large integrated systems share notes), the estimated Se/Sp do not apply to the population you will correct. This is selection bias dressed up as validation. - Imperfect or absent gold standard treated as truth. Using one claims rule to "validate" another, or an adjudication committee with modest inter-rater agreement, anchors the correction to a biased target. Report kappa, and use latent-class methods when no clean reference exists. - Assuming non-differentiality without checking. Surveillance, contact frequency, and coding intensity often differ by exposure. A non-differential matrix correction applied to genuinely differential misclassification can amplify bias. Validate Se/Sp within exposure strata when feasible, or carry differential scenarios in the PBA. - PPV-only validation when you need bias direction. A high PPV says flagged cases are real; it says nothing about the cases you missed. With low sensitivity and a rate or absolute-risk estimand, PPV alone cannot tell you which way, or how far, your estimate is off.

Data-source operational depth

- Claims (FFS vs MA vs commercial): Algorithms are built from ICD diagnosis codes, CPT/HCPCS procedures, NDCs, and place-of-service. Reference standard usually requires linkage to charts or a registry that the closed claims world does not contain, so PPV-only designs dominate. Real failure modes: (a) Medicare Advantage encounter data are under-captured relative to fee-for-service, so an algorithm validated in FFS over-states sensitivity when applied to MA person-time — restrict validation and analysis to comparable benefit types or validate within each. (b) ICD-9 to ICD-10 transition (Oct 2015) broke many code lists; a pre-2015 PPV does not carry forward. (c) Rule-in codes used for billing rather than confirmed disease (e.g., "rule-out MI" coded then refuted) inflate false positives; requiring an inpatient primary-position code plus a confirmatory procedure (e.g., troponin or revascularization) raises PPV at the cost of sensitivity. - EHR: Richer substrate (labs, vitals, notes) supports phenotype algorithms and even NLP, and notes can serve as part of the reference standard — but EHR capture is visit-driven and system-bounded: care delivered outside the system is invisible, so a "no event" can be unobserved rather than true, depressing apparent sensitivity. Local coding habits make EHR-derived Se/Sp poorly portable across institutions. - Registry: Often the reference standard itself (adjudicated stage, confirmed diagnosis) rather than the thing being validated; linking claims/EHR to a disease or death registry is the standard way to estimate Se/Sp because the registry captures truth the claims source cannot. - Linked claims-EHR-vital-records: Lets you adjudicate algorithm-negatives (needed for Se/Sp) using the partner source, but introduces linkage selection (only the linkable subset is validated) and date-discrepancy issues; validate the linkage rate and check that linked patients resemble the analytic cohort before generalizing the operating characteristics.

Worked claims example (outcome algorithm for acute MI, with correction)

Goal: validate and correct a claims algorithm for incident acute myocardial infarction (AMI) in a comparative drug-safety cohort. (1) Algorithm: an inpatient claim with an ICD-10-CM I21.x code in the primary position during a stay of >=1 day, with no AMI code in the 365-day washout (incident). (2) Eligibility for the analytic cohort: age >=18 and 365 days of continuous medical enrollment (FFS A/B, or commercial) before `index_date`, so absence of a prior AMI code is observed, not missing. (3) Sampling for validation: because we need a risk-ratio correction, draw a stratified random sample of both algorithm-positive and algorithm-negative person-records (weighting the rare positives up), within the linked subset where charts are obtainable. (4) Reference standard: chart adjudication using troponin dynamics and the universal MI definition; record `truth_flag`. (5) 2x2 and metrics: from `algorithm_flag` x `truth_flag` compute PPV with an exact (Clopper-Pearson) 95% CI; from the weighted positive and negative samples compute Se and Sp. (6) Correction: if Se=0.78, Sp=0.997 and the raw algorithm counted A_obs cases, the matrix-corrected true count is A_true = (A_obs - (1 - Sp) * N) / (Se - (1 - Sp)); apply within exposure arms and propagate uncertainty with a probabilistic bias analysis drawing Se/Sp from Beta distributions anchored to the validation 2x2. (7) Report per RECORD: the code list with versions, the sampling frame, the reference standard, the 2x2 counts, all four operating characteristics with CIs, and the corrected versus uncorrected effect estimate.

Worked example

Scenario

A research team is building a cohort of patients with Type 2 diabetes using an insurance claims database. Their algorithm flags any patient who has at least two outpatient diagnosis codes for Type 2 diabetes recorded on different dates. Before using this cohort to compare treatments, they need to know how accurate the algorithm is. They draw a random sample of 200 patient records — 100 the algorithm flagged as diabetes cases and 100 it did not flag — and have a clinician review each chart to record the true diagnosis.

Dataset

Results of chart review for the 200 sampled records, organized as a 2x2 table of algorithm decision versus gold-standard chart finding.

Steps

• Read the table: TP (true positives) = 80 patients the algorithm flagged who the chart confirmed do have diabetes; FP (false positives) = 20 patients the algorithm flagged who the chart showed do not have diabetes; FN (false negatives) = 20 patients the algorithm missed who the chart showed do have diabetes; TN (true negatives) = 80 patients the algorithm did not flag and the chart confirmed do not have diabetes.

• Compute PPV — ask 'of the 100 patients the algorithm flagged, how many are real cases?': PPV = TP / (TP + FP) = 80 / (80 + 20) = 80 / 100 = 0.80.

• Compute sensitivity — ask 'of the 100 patients who truly have diabetes, how many did the algorithm catch?': Sensitivity = TP / (TP + FN) = 80 / (80 + 20) = 80 / 100 = 0.80.

• Compute specificity — ask 'of the 100 patients who truly do not have diabetes, how many did the algorithm correctly leave un-flagged?': Specificity = TN / (TN + FP) = 80 / (80 + 20) = 80 / 100 = 0.80.

• Notice that each formula uses a different denominator: PPV divides by everyone the algorithm flagged (100); sensitivity divides by everyone who truly has the disease (100); specificity divides by everyone who truly does not have the disease (100). In real studies these three denominators are usually different numbers, which is why the three metrics can diverge dramatically.

Result

PPV = 0.80 (80 of 100 flagged patients are true cases), Sensitivity = 0.80 (the algorithm found 80 of the 100 true diabetes patients in the sample), Specificity = 0.80 (the algorithm correctly left un-flagged 80 of the 100 true non-cases). All three are 0.80 in this balanced example; in practice, a claims diabetes algorithm often has high PPV (few false positives) but lower sensitivity (missing patients who see out-of-network providers or pay out of pocket).

Runnable example

import numpy as np
from scipy import stats

def operating_characteristics(val):
    # 2x2 of algorithm_flag (rows) x truth_flag (cols). Use weights if a stratified sample was drawn.
    w = val.get("sample_weight", np.ones(len(val["algorithm_flag"])))
    a = np.asarray(val["algorithm_flag"]); t = np.asarray(val["truth_flag"])
    tp = (w[(a == 1) & (t == 1)]).sum()
    fp = (w[(a == 1) & (t == 0)]).sum()
    fn = (w[(a == 0) & (t == 1)]).sum()
    tn = (w[(a == 0) & (t == 0)]).sum()

    def exact_ci(num, den):
        # Clopper-Pearson exact binomial CI; rounds weighted counts to integers for the exact interval.
        num, den = int(round(num)), int(round(den))
        if den == 0:
            return (np.nan, np.nan, np.nan)
        lo = stats.beta.ppf(0.025, num, den - num + 1) if num > 0 else 0.0
        hi = stats.beta.ppf(0.975, num + 1, den - num) if num < den else 1.0
        return (num / den, lo, hi)

    return {
        "PPV": exact_ci(tp, tp + fp),   # NOT transportable: depends on target prevalence
        "NPV": exact_ci(tn, tn + fn),
        "Se":  exact_ci(tp, tp + fn),   # approximately transportable across populations
        "Sp":  exact_ci(tn, tn + fp),
    }

def matrix_correct_count(a_obs, n_total, se, sp):
    # Back-calculate the true number of cases from the algorithm-counted total under a given Se/Sp.
    # Apply this WITHIN each exposure arm to handle differential misclassification.
    denom = se - (1.0 - sp)
    if denom <= 0:
        raise ValueError("Se must exceed (1 - Sp); algorithm has no discriminating value.")
    a_true = (a_obs - (1.0 - sp) * n_total) / denom
    return max(a_true, 0.0)

def pba_corrected_count(a_obs, n_total, val, n_draws=20000, seed=1):
    # Probabilistic bias analysis: draw Se/Sp from Beta posteriors anchored to the validation 2x2,
    # propagate into the corrected case count, and summarize the simulation interval.
    rng = np.random.default_rng(seed)
    a = np.asarray(val["algorithm_flag"]); t = np.asarray(val["truth_flag"])
    tp = int(((a == 1) & (t == 1)).sum()); fn = int(((a == 0) & (t == 1)).sum())
    tn = int(((a == 0) & (t == 0)).sum()); fp = int(((a == 1) & (t == 0)).sum())
    se_draws = rng.beta(tp + 0.5, fn + 0.5, n_draws)   # Jeffreys prior on Se
    sp_draws = rng.beta(tn + 0.5, fp + 0.5, n_draws)   # Jeffreys prior on Sp
    out = np.array([matrix_correct_count(a_obs, n_total, s, p)
                    for s, p in zip(se_draws, sp_draws)
                    if (s - (1.0 - p)) > 0])
    return {"median": float(np.median(out)),
            "ci95": (float(np.percentile(out, 2.5)), float(np.percentile(out, 97.5)))}

op_char <- function(val) {
  w <- if (!is.null(val$sample_weight)) val$sample_weight else rep(1, nrow(val))
  a <- val$algorithm_flag; t <- val$truth_flag
  tp <- sum(w[a == 1 & t == 1]); fp <- sum(w[a == 1 & t == 0])
  fn <- sum(w[a == 0 & t == 1]); tn <- sum(w[a == 0 & t == 0])

  exact_ci <- function(num, den) {           # Clopper-Pearson exact binomial CI
    num <- round(num); den <- round(den)
    if (den == 0) return(c(est = NA, lo = NA, hi = NA))
    bt <- binom.test(num, den)
    c(est = num / den, lo = bt$conf.int[1], hi = bt$conf.int[2])
  }
  list(PPV = exact_ci(tp, tp + fp),   # NOT transportable across populations
       NPV = exact_ci(tn, tn + fn),
       Se  = exact_ci(tp, tp + fn),   # approximately transportable
       Sp  = exact_ci(tn, tn + fp))
}

matrix_correct_count <- function(a_obs, n_total, se, sp) {
  denom <- se - (1 - sp)
  if (denom <= 0) stop("Se must exceed (1 - Sp); algorithm does not discriminate.")
  max((a_obs - (1 - sp) * n_total) / denom, 0)
}

pba_corrected_count <- function(a_obs, n_total, val, n_draws = 20000L, seed = 1L) {
  set.seed(seed)
  a <- val$algorithm_flag; t <- val$truth_flag
  tp <- sum(a == 1 & t == 1); fn <- sum(a == 0 & t == 1)
  tn <- sum(a == 0 & t == 0); fp <- sum(a == 1 & t == 0)
  se <- rbeta(n_draws, tp + 0.5, fn + 0.5)   # Jeffreys prior on Se
  sp <- rbeta(n_draws, tn + 0.5, fp + 0.5)   # Jeffreys prior on Sp
  keep <- (se - (1 - sp)) > 0
  out <- mapply(matrix_correct_count, se = se[keep], sp = sp[keep],
                MoreArgs = list(a_obs = a_obs, n_total = n_total))
  list(median = median(out),
       ci95 = quantile(out, c(0.025, 0.975), names = FALSE))
}