Diagnostic Accuracy Study

In plain language

A diagnostic accuracy study asks a simple question: how often does a test (or a claims/EHR rule that flags who has a disease) get the right answer when you check it against a trusted reference, like a chart review? You sort every patient into one of four buckets — correctly flagged sick, wrongly flagged sick, correctly cleared, wrongly cleared — and count them in a 2x2 table. From those four counts you can describe the test in plain terms: how many truly-sick people it catches, and how many truly-healthy people it correctly clears. One honest caveat: a single 'percent correct' number can look great and still hide a test that misses almost everyone who is actually sick.

A diagnostic accuracy study measures the agreement between an index test (a biomarker, imaging read, clinical rule, or — in real-world evidence — a claims/EHR case-finding algorithm) and a reference standard that is taken as the best available classification of true disease status. Each subject contributes a 2x2 cross-classification (index positive/negative x reference positive/negative), and the design summarizes the test's operating characteristics: sensitivity (Pr[index+ | disease+]), specificity (Pr[index- | disease-]), positive and negative predictive value (PPV/NPV), and the positive/negative likelihood ratios (LR+ = sens/(1-spec), LR- = (1-sens)/spec). It is the foundational design behind validating any RWE phenotype before that phenotype is trusted to define an exposure or outcome.

Core conceptual distinction

The single most consequential teaching point is that sensitivity and specificity are properties of the test conditional on true disease status and are (to first order) invariant to disease prevalence, whereas PPV and NPV are prevalence-dependent and therefore travel poorly across populations. A claims algorithm validated at 90% PPV in a high-prevalence specialty registry can collapse to 50% PPV in a low-prevalence general population even though its sensitivity and specificity are unchanged — PPV = (sens x prev) / (sens x prev + (1-spec) x (1-prev)). Likelihood ratios are the prevalence-stable bridge: they update pre-test odds to post-test odds via Bayes (post-test odds = pre-test odds x LR), so a single LR set transports to any prevalence. The second distinction is the estimand under the sampling scheme: a cohort/cross-sectional sample (consecutive eligible subjects) identifies sensitivity, specificity, PPV, and NPV directly; a case-control (two-gate) sample — selecting known cases and known non-cases — identifies sensitivity and specificity but not PPV/NPV, because the case:non-case ratio is fixed by design rather than reflecting prevalence. A third distinction separates a diagnostic accuracy study (test vs reference, no follow-up needed) from a prognostic/predictive study (baseline test vs future outcome over time).

Pros, cons, and trade-offs

- vs validating a phenotype by PPV alone (chart-review of algorithm-positives only): The full diagnostic accuracy study estimates sensitivity and specificity, which a PPV-only review cannot — you cannot detect under-capture (false negatives) by reviewing only test-positives. Cost: estimating sensitivity requires sampling the reference-positive or test-negative space, which is expensive when disease is rare. Prefer the full study when the algorithm defines an outcome whose completeness matters (e.g., differential misclassification across arms); a focused high-PPV review can suffice when the algorithm only needs to confirm cases for a positive predictive purpose (see claims-outcome-algorithm-ppv-sensitivity-rwe). - vs misclassification bias correction (quantitative adjustment of the effect estimate): A diagnostic accuracy study produces the bias parameters (sensitivity/specificity, ideally by arm) that misclassification-bias-correction and quantitative-bias-analysis methods consume. The accuracy study is descriptive of the measurement; the correction propagates that measurement uncertainty into the comparative estimate. Use them together — an accuracy study without a downstream correction leaves known bias on the table. - vs treating the index test as a gold standard (no validation): Default RWE practice often assumes the algorithm is perfect. That is defensible only for highly specific procedure/anchor codes; for most diagnosis-based phenotypes, unvalidated use risks non-differential bias toward the null (or, worse, differential bias of unknown direction). Prefer at least a validation substudy whenever the algorithm is novel, transported to a new data source, or central to the primary estimand.

When to use

Validating a new claims/EHR case-finding algorithm before deploying it as an exposure or outcome definition; benchmarking a biomarker, score, or imaging read against a reference standard; supporting a HEDIS/PQA-style quality measure or an FDA fit-for-purpose data assessment that requires documented outcome misclassification parameters; quantifying the bias parameters that a downstream quantitative bias analysis will use; comparing two competing algorithms head-to-head on the same validation sample (paired design, McNemar-based comparison).

When NOT to use — and when it is actively misleading or dangerous

. - No genuinely independent reference standard exists. If the reference is built partly from the same data feeding the index test (e.g., chart review by an adjudicator who can see the billing codes the algorithm used), incorporation bias inflates apparent accuracy. The reference must be ascertained blind to the index result. - The reference standard is itself imperfect (no true gold standard). Comparing an index test to a noisy reference can make a better index test look worse when the two err on different subjects; naive 2x2 accuracy is biased and an imperfect-reference (latent-class or explicit-correction) model is required. Reporting raw sensitivity/specificity against a known-bad reference is misleading. - Reference verification depends on the index test result (verification / work-up bias). If only index-positive (or sicker-looking) subjects get the reference standard — the norm when chart review is triggered by the algorithm firing — uncorrected sensitivity is overestimated and specificity underestimated. This is the single most common, and most dangerous, error in RWE phenotype validation; it requires a Begg-Greenes-type correction or a two-stage design with known sampling probabilities. - Spectrum mismatch. Accuracy estimated in a referred, severe, or clinically extreme sample (clear cases vs healthy controls) does not transport to the borderline, early-stage, comorbid patients on whom the test is actually used (spectrum bias). A test that looks excellent in a two-gate case-control sample can be useless in consecutive practice. - Reporting PPV/NPV from a case-control sample, or transporting PPV across prevalences. Both are formally invalid; report sensitivity/specificity/LRs and re-derive predictive values at the target prevalence instead.

Data-source operational depth

- Claims (FFS vs MA vs commercial): The index test is the algorithm (e.g., 1 inpatient OR 2 outpatient diagnosis codes >=30 days apart within a defined window). The reference standard is usually chart review obtained via linkage, so accuracy can be estimated only on the linkable subset — a selection that must be argued not to be differential. Medicare Advantage encounter data historically under-captures procedures and pharmacy relative to fee-for-service, so an algorithm validated in FFS can have different sensitivity in MA person-time; validate within benefit type and never pool blindly (see medicare-ffs-ma-commercial-claims-differences-rwe). Code-set version drift (ICD-9 to ICD-10) silently changes sensitivity over calendar time. - EHR: A richer reference is available in-system (labs, pathology, notes via NLP), but capture is visit-driven and fragmented across systems, so a "false negative" may be care delivered elsewhere rather than true absence of disease — define the observation window and treat out-of-system care as potential misclassification, not truth. - Registry: Often the reference standard itself (adjudicated cancer stage, confirmed MI), making it ideal for validating claims/EHR algorithms — but registries capture a selected, often more severe spectrum, so accuracy can be optimistic relative to community practice. - Linked claims-EHR-registry: The strongest substrate (algorithm from claims, reference from registry/EHR, completeness from linkage) but introduces linkage selection and date discrepancies between service, fill, and adjudication dates that must be reconciled before the 2x2 is built.

Worked claims example

Goal: validate a claims algorithm for acute myocardial infarction (AMI) to use it as a study outcome. (1) Source and eligibility: adults with >=12 months of continuous Medicare FFS Parts A/B enrollment (so absence of a code is observed, not missing) and no AMI code in a 12-month clean washout, so captured events are incident. (2) Index test (algorithm): a primary-position ICD-10 I21.x on an inpatient claim with length of stay >=1 day — the standard high-specificity AMI rule. (3) Reference standard: hospital-chart review against the Fourth Universal Definition of MI (troponin + clinical criteria), adjudicated by reviewers blinded to the billing codes to avoid incorporation bias. (4) Sampling: because chart pulls are costly, use a two-stage stratified design — review all (or a known fraction f1 of) algorithm-positives to estimate PPV, AND a known fraction f2 of a reference-positive sampling frame built from troponin-lab-flagged admissions to estimate sensitivity; record f1 and f2 so estimates can be reweighted to the cohort. (5) Build the 2x2 on the linkable, chart-available subset, weighting by inverse sampling fraction so cells reflect the source cohort, not the review sample. (6) Report sensitivity and specificity (prevalence-stable) with exact (Clopper-Pearson) 95% CIs, plus PPV/NPV at the cohort's own AMI prevalence, and LR+ / LR-. (7) Flag verification bias explicitly: because the reference was preferentially obtained where troponin was drawn (correlated with the index test firing), apply a Begg-Greenes correction or report the corrected sensitivity/specificity. (8) Carry the resulting (sens, spec) — ideally estimated separately by exposure arm — into a misclassification-bias-correction step so the final comparative effect is adjusted for outcome misclassification rather than reported as if the algorithm were perfect.

Worked example

Scenario

We built a claims rule to flag patients with a rare disease, and we want to know how good it is. We took 1,000 patients, ran the rule on each one (that is our index test), and also got a blinded chart review on every patient (that is our reference standard, treated as the truth). The disease is uncommon: only 50 of the 1,000 patients truly have it. We will sort all 1,000 into a 2x2 table, compute the overall accuracy, and then show why that single number can fool us.

Dataset

The 2x2 confusion table an analyst would build: index test (claims rule) result crossed against the reference standard (chart review). Each patient lands in exactly one cell, and the four cells sum to N = 1,000.

Steps

• Read the four cells. TP = 40 patients the rule flagged who truly had the disease. TN = 855 the rule cleared who truly did not. FP = 95 the rule wrongly flagged. FN = 10 truly-sick patients the rule missed.

• Confirm the table closes: TP + FP + FN + TN = 40 + 95 + 10 + 855 = 1000 = N. The 50 truly diseased (TP + FN) and 950 truly healthy (FP + TN) match the column totals.

• Compute overall accuracy = the share of patients the rule got right = (TP + TN) / N = (40 + 855) / 1000 = 895 / 1000 = 0.895.

• That 89.5% sounds strong, but the disease is rare — only 5% of patients have it (class imbalance). Picture a lazy rule that simply calls EVERY patient negative. It would be right on all 950 healthy patients and wrong only on the 50 sick ones, giving accuracy = 950 / 1000 = 0.95 — HIGHER than our real rule, while catching zero true cases.

• So accuracy alone hides the failure mode. Split it into the two numbers that do not get fooled by the rare/common mix. Sensitivity = among the truly sick, the share caught = TP / (TP + FN) = 40 / 50 = 0.80. Specificity = among the truly healthy, the share cleared = TN / (TN + FP) = 855 / 950 = 0.90.

• Now the picture is honest: the rule catches 80% of true cases and correctly clears 90% of non-cases. The lazy 'all-negative' rule would have specificity 1.00 but sensitivity 0.00 — and accuracy alone would have rewarded it. Always report sensitivity and specificity, not just percent correct.

Result

Overall accuracy = (TP + TN) / N = (40 + 855) / 1000 = 0.895 (89.5%). Sensitivity = 40 / 50 = 0.80 (80%); specificity = 855 / 950 = 0.90 (90%). The catch: because only 5% of patients are diseased, a useless rule that calls everyone negative scores accuracy = 950 / 1000 = 0.95 — beating our real rule on accuracy while having sensitivity = 0. Under class imbalance, accuracy is misleading; sensitivity and specificity are not.

Runnable example

import numpy as np
import pandas as pd
from scipy.stats import beta

def clopper_pearson(k: float, n: float, alpha: float = 0.05):
    # Exact binomial CI for a proportion k/n (works on integer counts).
    k, n = int(round(k)), int(round(n))
    lo = 0.0 if k == 0 else beta.ppf(alpha / 2, k, n - k + 1)
    hi = 1.0 if k == n else beta.ppf(1 - alpha / 2, k + 1, n - k)
    return k / n if n else np.nan, lo, hi

def diagnostic_accuracy(cohort: pd.DataFrame) -> dict:
    # Restrict to subjects with a reference result (verified subset).
    v = cohort.dropna(subset=["ref_pos"]).copy()
    w = v["samp_wt"].to_numpy()  # inverse-probability weights reweight to the source cohort

    # Weighted 2x2 cells.
    tp = float((w * ((v.index_pos == 1) & (v.ref_pos == 1))).sum())
    fp = float((w * ((v.index_pos == 1) & (v.ref_pos == 0))).sum())
    fn = float((w * ((v.index_pos == 0) & (v.ref_pos == 1))).sum())
    tn = float((w * ((v.index_pos == 0) & (v.ref_pos == 0))).sum())

    sens, sens_lo, sens_hi = clopper_pearson(tp, tp + fn)
    spec, spec_lo, spec_hi = clopper_pearson(tn, tn + fp)
    ppv,  ppv_lo,  ppv_hi  = clopper_pearson(tp, tp + fp)   # prevalence-dependent
    npv,  npv_lo,  npv_hi  = clopper_pearson(tn, tn + fn)   # prevalence-dependent
    prevalence = (tp + fn) / (tp + fp + fn + tn)
    lr_pos = sens / (1 - spec) if spec < 1 else np.inf
    lr_neg = (1 - sens) / spec if spec > 0 else np.inf
    return {
        "cells": {"tp": tp, "fp": fp, "fn": fn, "tn": tn},
        "prevalence": prevalence,
        "sensitivity": (sens, sens_lo, sens_hi),
        "specificity": (spec, spec_lo, spec_hi),
        "ppv": (ppv, ppv_lo, ppv_hi),
        "npv": (npv, npv_lo, npv_hi),
        "lr_positive": lr_pos,
        "lr_negative": lr_neg,
    }

def ppv_at_prevalence(sens: float, spec: float, prev: float) -> float:
    # Re-derive PPV at any target prevalence (transport via Bayes); sens/spec are stable, PPV is not.
    return (sens * prev) / (sens * prev + (1 - spec) * (1 - prev))

diagnostic_accuracy <- function(cohort) {
  v <- cohort[!is.na(cohort$ref_pos), ]
  w <- v$samp_wt  # inverse-probability weights reweight verified subset to the source cohort

  tp <- sum(w * (v$index_pos == 1 & v$ref_pos == 1))
  fp <- sum(w * (v$index_pos == 1 & v$ref_pos == 0))
  fn <- sum(w * (v$index_pos == 0 & v$ref_pos == 1))
  tn <- sum(w * (v$index_pos == 0 & v$ref_pos == 0))

  ci <- function(k, n) {                # exact Clopper-Pearson CI on rounded counts
    bt <- binom.test(round(k), round(n))
    c(est = unname(bt$estimate), lo = bt$conf.int[1], hi = bt$conf.int[2])
  }
  sens <- ci(tp, tp + fn); spec <- ci(tn, tn + fp)
  ppv  <- ci(tp, tp + fp); npv  <- ci(tn, tn + fn)   # PPV/NPV are prevalence-dependent
  list(
    cells       = c(tp = tp, fp = fp, fn = fn, tn = tn),
    prevalence  = (tp + fn) / (tp + fp + fn + tn),
    sensitivity = sens, specificity = spec, ppv = ppv, npv = npv,
    lr_positive = sens["est"] / (1 - spec["est"]),
    lr_negative = (1 - sens["est"]) / spec["est"]
  )
}

ppv_at_prevalence <- function(sens, spec, prev) {
  (sens * prev) / (sens * prev + (1 - spec) * (1 - prev))
}