Diagnostic Accuracy Study

A diagnostic accuracy study measures the agreement between an index test (a biomarker, imaging read, clinical rule, or — in real-world evidence — a claims/EHR case-finding algorithm) and a reference standard that is taken as the best available classification of true disease status. Each subject contributes a 2x2 cross-classification (index positive/negative x reference positive/negative), and the design summarizes the test's operating characteristics: sensitivity (Pr[index+ | disease+]), specificity (Pr[index- | disease-]), positive and negative predictive value (PPV/NPV), and the positive/negative likelihood ratios (LR+ = sens/(1-spec), LR- = (1-sens)/spec).

It is the foundational design behind validating any RWE phenotype before that phenotype is trusted to define an exposure or outcome.

Core conceptual distinction

The single most consequential teaching point is that sensitivity and specificity are properties of the test conditional on true disease status and are (to first order) invariant to disease prevalence, whereas PPV and NPV are prevalence-dependent and therefore travel poorly across populations. A claims algorithm validated at 90% PPV in a high-prevalence specialty registry can collapse to 50% PPV in a low-prevalence general population even though its sensitivity and specificity are unchanged — PPV = (sens x prev) / (sens x prev + (1-spec) x (1-prev)). Likelihood ratios are the prevalence-stable bridge: they update pre-test odds to post-test odds via Bayes (post-test odds = pre-test odds x LR), so a single LR set transports to any prevalence.

The second distinction is the estimand under the sampling scheme: a cohort/cross-sectional sample (consecutive eligible subjects) identifies sensitivity, specificity, PPV, and NPV directly; a case-control (two-gate) sample — selecting known cases and known non-cases — identifies sensitivity and specificity but not PPV/NPV, because the case:non-case ratio is fixed by design rather than reflecting prevalence.

A third distinction separates a diagnostic accuracy study (test vs reference, no follow-up needed) from a prognostic/predictive study (baseline test vs future outcome over time).

Pros, cons, and trade-offs

• vs validating a phenotype by PPV alone (chart-review of algorithm-positives only): The full diagnostic accuracy study estimates sensitivity and specificity, which a PPV-only review cannot — you cannot detect under-capture (false negatives) by reviewing only test-positives. Cost: estimating sensitivity requires sampling the reference-positive or test-negative space, which is expensive when disease is rare. Prefer the full study when the algorithm defines an outcome whose completeness matters (e.g., differential misclassification across arms); a focused high-PPV review can suffice when the algorithm only needs to confirm cases for a positive predictive purpose (see claims-outcome-algorithm-ppv-sensitivity-rwe).
• vs misclassification bias correction (quantitative adjustment of the effect estimate): A diagnostic accuracy study produces the bias parameters (sensitivity/specificity, ideally by arm) that misclassification-bias-correction and quantitative-bias-analysis methods consume. The accuracy study is descriptive of the measurement; the correction propagates that measurement uncertainty into the comparative estimate. Use them together — an accuracy study without a downstream correction leaves known bias on the table.
• vs treating the index test as a gold standard (no validation): Default RWE practice often assumes the algorithm is perfect. That is defensible only for highly specific procedure/anchor codes; for most diagnosis-based phenotypes, unvalidated use risks non-differential bias toward the null (or, worse, differential bias of unknown direction). Prefer at least a validation substudy whenever the algorithm is novel, transported to a new data source, or central to the primary estimand.

When NOT to use — and when it is actively misleading or dangerous

.

• No genuinely independent reference standard exists. If the reference is built partly from the same data feeding the index test (e.g., chart review by an adjudicator who can see the billing codes the algorithm used), incorporation bias inflates apparent accuracy. The reference must be ascertained blind to the index result.
• The reference standard is itself imperfect (no true gold standard). Comparing an index test to a noisy reference can make a better index test look worse when the two err on different subjects; naive 2x2 accuracy is biased and an imperfect-reference (latent-class or explicit-correction) model is required. Reporting raw sensitivity/specificity against a known-bad reference is misleading.
• Reference verification depends on the index test result (verification / work-up bias). If only index-positive (or sicker-looking) subjects get the reference standard — the norm when chart review is triggered by the algorithm firing — uncorrected sensitivity is overestimated and specificity underestimated. This is the single most common, and most dangerous, error in RWE phenotype validation; it requires a Begg-Greenes-type correction or a two-stage design with known sampling probabilities.
• Spectrum mismatch. Accuracy estimated in a referred, severe, or clinically extreme sample (clear cases vs healthy controls) does not transport to the borderline, early-stage, comorbid patients on whom the test is actually used (spectrum bias). A test that looks excellent in a two-gate case-control sample can be useless in consecutive practice.
• Reporting PPV/NPV from a case-control sample, or transporting PPV across prevalences. Both are formally invalid; report sensitivity/specificity/LRs and re-derive predictive values at the target prevalence instead.

Data-source operational depth

• Claims (FFS vs MA vs commercial): The index test is the algorithm (e.g., 1 inpatient OR 2 outpatient diagnosis codes >=30 days apart within a defined window). The reference standard is usually chart review obtained via linkage, so accuracy can be estimated only on the linkable subset — a selection that must be argued not to be differential. Medicare Advantage encounter data historically under-captures procedures and pharmacy relative to fee-for-service, so an algorithm validated in FFS can have different sensitivity in MA person-time; validate within benefit type and never pool blindly (see medicare-ffs-ma-commercial-claims-differences-rwe). Code-set version drift (ICD-9 to ICD-10) silently changes sensitivity over calendar time.
• EHR: A richer reference is available in-system (labs, pathology, notes via NLP), but capture is visit-driven and fragmented across systems, so a "false negative" may be care delivered elsewhere rather than true absence of disease — define the observation window and treat out-of-system care as potential misclassification, not truth.
• Registry: Often the reference standard itself (adjudicated cancer stage, confirmed MI), making it ideal for validating claims/EHR algorithms — but registries capture a selected, often more severe spectrum, so accuracy can be optimistic relative to community practice.
• Linked claims-EHR-registry: The strongest substrate (algorithm from claims, reference from registry/EHR, completeness from linkage) but introduces linkage selection and date discrepancies between service, fill, and adjudication dates that must be reconciled before the 2x2 is built.

Worked claims example

Goal: validate a claims algorithm for acute myocardial infarction (AMI) to use it as a study outcome.

1. Source and eligibility: adults with >=12 months of continuous Medicare FFS Parts A/B enrollment (so absence of a code is observed, not missing) and no AMI code in a 12-month clean washout, so captured events are incident.
2. Index test (algorithm): a primary-position ICD-10 I21.x on an inpatient claim with length of stay >=1 day — the standard high-specificity AMI rule.
3. Reference standard: hospital-chart review against the Fourth Universal Definition of MI (troponin + clinical criteria), adjudicated by reviewers blinded to the billing codes to avoid incorporation bias.
4. Sampling: because chart pulls are costly, use a two-stage stratified design — review all (or a known fraction f1 of) algorithm-positives to estimate PPV, AND a known fraction f2 of a reference-positive sampling frame built from troponin-lab-flagged admissions to estimate sensitivity; record f1 and f2 so estimates can be reweighted to the cohort.
5. Build the 2x2 on the linkable, chart-available subset, weighting by inverse sampling fraction so cells reflect the source cohort, not the review sample.
6. Report sensitivity and specificity (prevalence-stable) with exact (Clopper-Pearson) 95% CIs, plus PPV/NPV at the cohort's own AMI prevalence, and LR+ / LR-.
7. Flag verification bias explicitly: because the reference was preferentially obtained where troponin was drawn (correlated with the index test firing), apply a Begg-Greenes correction or report the corrected sensitivity/specificity.
8. Carry the resulting (sens, spec) — ideally estimated separately by exposure arm — into a misclassification-bias-correction step so the final comparative effect is adjusted for outcome misclassification rather than reported as if the algorithm were perfect.

Decision diagram

flowchart TD
  Src[Eligible subjects with the indication<br/>continuous enrollment / observable time] --> Idx[Apply index test<br/>= claims/EHR case-finding algorithm]
  Idx --> Ref{Reference standard obtained?}
  Ref -- All subjects (single-gate cohort) --> Tab[Cross-classify index x reference]
  Ref -- Only a subset, often triggered by index+ --> VB[VERIFICATION / WORK-UP BIAS<br/>weight by known sampling probability<br/>or Begg-Greenes correction]
  VB --> Tab
  Tab --> M[Estimate sensitivity & specificity<br/>prevalence-stable, with exact CIs]
  Tab --> P[Estimate PPV / NPV<br/>prevalence-DEPENDENT: report at cohort prevalence]
  M --> LR[Likelihood ratios -> transport via Bayes]
  M --> BC[Carry sens/spec into misclassification bias correction]

flowchart LR
  subgraph TwoByTwo [2x2 against the reference standard]
    TP[TP: index+ / ref+] --- FP[FP: index+ / ref-]
    FN[FN: index- / ref+] --- TN[TN: index- / ref-]
  end
  TP --> SENS[Sensitivity = TP / TP+FN<br/>prevalence-stable]
  FN --> SENS
  TN --> SPEC[Specificity = TN / TN+FP<br/>prevalence-stable]
  FP --> SPEC
  TP --> PPV[PPV = TP / TP+FP<br/>prevalence-DEPENDENT]
  FP --> PPV
  TN --> NPV[NPV = TN / TN+FN<br/>prevalence-DEPENDENT]
  FN --> NPV

Worked example

Scenario

We built a claims rule to flag patients with a rare disease, and we want to know how good it is. We took 1,000 patients, ran the rule on each one (that is our index test), and also got a blinded chart review on every patient (that is our reference standard, treated as the truth). The disease is uncommon: only 50 of the 1,000 patients truly have it. We will sort all 1,000 into a 2x2 table, compute the overall accuracy, and then show why that single number can fool us.

Dataset

The 2x2 confusion table an analyst would build: index test (claims rule) result crossed against the reference standard (chart review). Each patient lands in exactly one cell, and the four cells sum to N = 1,000.

Steps

Result

Overall accuracy = (TP + TN) / N = (40 + 855) / 1000 = 0.895 (89.5%). Sensitivity = 40 / 50 = 0.80 (80%); specificity = 855 / 950 = 0.90 (90%). The catch: because only 5% of patients are diseased, a useless rule that calls everyone negative scores accuracy = 950 / 1000 = 0.95 — beating our real rule on accuracy while having sensitivity = 0. Under class imbalance, accuracy is misleading; sensitivity and specificity are not.

Trade-offs

Runnable example

import numpy as np
import pandas as pd
from scipy.stats import beta

def clopper_pearson(k: float, n: float, alpha: float = 0.05):
    # Exact binomial CI for a proportion k/n (works on integer counts).
    k, n = int(round(k)), int(round(n))
    lo = 0.0 if k == 0 else beta.ppf(alpha / 2, k, n - k + 1)
    hi = 1.0 if k == n else beta.ppf(1 - alpha / 2, k + 1, n - k)
    return k / n if n else np.nan, lo, hi

def diagnostic_accuracy(cohort: pd.DataFrame) -> dict:
    # Restrict to subjects with a reference result (verified subset).
    v = cohort.dropna(subset=["ref_pos"]).copy()
    w = v["samp_wt"].to_numpy()  # inverse-probability weights reweight to the source cohort

    # Weighted 2x2 cells.
    tp = float((w * ((v.index_pos == 1) & (v.ref_pos == 1))).sum())
    fp = float((w * ((v.index_pos == 1) & (v.ref_pos == 0))).sum())
    fn = float((w * ((v.index_pos == 0) & (v.ref_pos == 1))).sum())
    tn = float((w * ((v.index_pos == 0) & (v.ref_pos == 0))).sum())

    sens, sens_lo, sens_hi = clopper_pearson(tp, tp + fn)
    spec, spec_lo, spec_hi = clopper_pearson(tn, tn + fp)
    ppv,  ppv_lo,  ppv_hi  = clopper_pearson(tp, tp + fp)   # prevalence-dependent
    npv,  npv_lo,  npv_hi  = clopper_pearson(tn, tn + fn)   # prevalence-dependent
    prevalence = (tp + fn) / (tp + fp + fn + tn)
    lr_pos = sens / (1 - spec) if spec < 1 else np.inf
    lr_neg = (1 - sens) / spec if spec > 0 else np.inf
    return {
        "cells": {"tp": tp, "fp": fp, "fn": fn, "tn": tn},
        "prevalence": prevalence,
        "sensitivity": (sens, sens_lo, sens_hi),
        "specificity": (spec, spec_lo, spec_hi),
        "ppv": (ppv, ppv_lo, ppv_hi),
        "npv": (npv, npv_lo, npv_hi),
        "lr_positive": lr_pos,
        "lr_negative": lr_neg,
    }

def ppv_at_prevalence(sens: float, spec: float, prev: float) -> float:
    # Re-derive PPV at any target prevalence (transport via Bayes); sens/spec are stable, PPV is not.
    return (sens * prev) / (sens * prev + (1 - spec) * (1 - prev))

diagnostic_accuracy <- function(cohort) {
  v <- cohort[!is.na(cohort$ref_pos), ]
  w <- v$samp_wt  # inverse-probability weights reweight verified subset to the source cohort

  tp <- sum(w * (v$index_pos == 1 & v$ref_pos == 1))
  fp <- sum(w * (v$index_pos == 1 & v$ref_pos == 0))
  fn <- sum(w * (v$index_pos == 0 & v$ref_pos == 1))
  tn <- sum(w * (v$index_pos == 0 & v$ref_pos == 0))

  ci <- function(k, n) {                # exact Clopper-Pearson CI on rounded counts
    bt <- binom.test(round(k), round(n))
    c(est = unname(bt$estimate), lo = bt$conf.int[1], hi = bt$conf.int[2])
  }
  sens <- ci(tp, tp + fn); spec <- ci(tn, tn + fp)
  ppv  <- ci(tp, tp + fp); npv  <- ci(tn, tn + fn)   # PPV/NPV are prevalence-dependent
  list(
    cells       = c(tp = tp, fp = fp, fn = fn, tn = tn),
    prevalence  = (tp + fn) / (tp + fp + fn + tn),
    sensitivity = sens, specificity = spec, ppv = ppv, npv = npv,
    lr_positive = sens["est"] / (1 - spec["est"]),
    lr_negative = (1 - sens["est"]) / spec["est"]
  )
}

ppv_at_prevalence <- function(sens, spec, prev) {
  (sens * prev) / (sens * prev + (1 - spec) * (1 - prev))
}

/* Stage 1: weighted 2x2 on the verified subset (ref_pos not missing). */
proc sql;
  create table cells as
  select sum(samp_wt * (index_pos=1 and ref_pos=1)) as tp,
         sum(samp_wt * (index_pos=1 and ref_pos=0)) as fp,
         sum(samp_wt * (index_pos=0 and ref_pos=1)) as fn,
         sum(samp_wt * (index_pos=0 and ref_pos=0)) as tn
  from work.cohort
  where ref_pos is not null;
quit;

/* Stage 2a: exact (Clopper-Pearson) CIs for sensitivity (within reference-positive stratum)
   and specificity (within reference-negative stratum). WEIGHT applies the sampling weights. */
proc freq data=work.cohort(where=(ref_pos=1)) order=data;
  tables index_pos / binomial(level='1') alpha=0.05;   /* P(index+ | ref+) = sensitivity */
  weight samp_wt;
  exact binomial;
run;

proc freq data=work.cohort(where=(ref_pos=0)) order=data;
  tables index_pos / binomial(level='0') alpha=0.05;   /* P(index- | ref-) = specificity */
  weight samp_wt;
  exact binomial;
run;

/* Stage 2b: predictive values (prevalence-dependent) and likelihood ratios from the 2x2. */
data accuracy;
  set cells;
  sensitivity = tp / (tp + fn);
  specificity = tn / (tn + fp);
  ppv         = tp / (tp + fp);
  npv         = tn / (tn + fn);
  prevalence  = (tp + fn) / (tp + fp + fn + tn);
  lr_positive = sensitivity / (1 - specificity);
  lr_negative = (1 - sensitivity) / specificity;
run;

Citations

1. [1]Sackett DL, Haynes RB. The architecture of diagnostic research. BMJ. 2002;324(7336):539-541.
2. [2]Ransohoff DF, Feinstein AR. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. New England Journal of Medicine. 1978;299(17):926-930.
3. [3]Begg CB, Greenes RA. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics. 1983;39(1):207-215.
4. [4]Deeks JJ, Altman DG. Diagnostic tests 4: likelihood ratios. BMJ. 2004;329(7458):168-169.

1. [5]Whiting PF, Rutjes AWS, Westwood ME, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Annals of Internal Medicine. 2011;155(8):529-536.

1. [6]Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527.