Sensitivity and Specificity

In plain language

Sensitivity and specificity tell you how well a yes/no test or a code-based rule matches the real truth, when you already know the truth from a careful source like a chart review. Sensitivity is the share of people who truly have the condition that the test correctly flags as positive; specificity is the share of people who truly don't have it that the test correctly clears as negative. A handy property: both numbers stay the same no matter how common the disease is, which is why they are the go-to summary when you are checking how good an algorithm is. One honest caveat: they say nothing about how trustworthy a single positive flag is in your data, because that depends on how common the disease is.

Sensitivity

and specificity characterize how well a test, code-based algorithm, or computable phenotype recovers a true binary status when judged against a reference ("gold") standard. Lay the 2x2 table out with truth in columns and the test in rows: true positives (TP), false positives (FP), false negatives (FN), true negatives (TN). Sensitivity = TP / (TP + FN) is the conditional probability of a positive test given the patient truly has the condition (the true-positive rate, P(T+|D+)); specificity = TN / (TN + FP) is the conditional probability of a negative test given the patient truly does not (the true-negative rate, P(T−|D−)). Their complements are the false-negative rate (1 − sensitivity) and false-positive rate (1 − specificity). Because each is conditioned on the true status, both are properties of the test on a fixed disease spectrum and do not depend on how common the disease is — this prevalence-invariance is the single most important property distinguishing them from predictive values.

Core idea

Sensitivity and specificity answer the "test-evaluation" question — given the truth, how does the test behave? — which is the question a method developer asks. They are the natural targets of a validation study that samples on true status (e.g., chart-confirmed cases and confirmed non-cases) and applies the algorithm to each. The contrast is with predictive values (PPV/NPV), which answer the clinician's question — given the test result, what is the truth? — and which depend on prevalence. In real-world data the reference standard is usually adjudicated chart review, registry linkage, or a validated lab result; the "test" is a claims/EHR code algorithm (e.g., "1 inpatient or 2 outpatient ICD-10 codes for the condition within 12 months"). Sensitivity and specificity are then operating characteristics of that algorithm, and they propagate directly into outcome misclassification: a non-differential algorithm with imperfect sensitivity/specificity biases a measured risk ratio toward the null in a predictable way, and they are the inputs to quantitative-bias-analysis correction.

Prevalence invariance and its limits

Holding the case-mix (disease spectrum) fixed, doubling prevalence changes the column totals but not the within-column proportions, so sensitivity and specificity are unchanged. The caveat is spectrum effect: sensitivity tends to be higher in severe, advanced, or clearly symptomatic cases and lower in mild or early disease, and specificity varies with the comorbidity profile of the non-cases. Estimates from a tertiary-care validation set therefore may not transport to a community claims population whose case spectrum differs. Invariance to prevalence is mathematical; transportability across spectra is an empirical claim that must be checked.

Non-differential vs differential misclassification

When the algorithm's sensitivity and specificity are the same in the exposed and unexposed (non-differential, independent of exposure), the resulting outcome misclassification biases a risk ratio or odds ratio toward the null on average (for a binary outcome with independent errors), attenuating but not reversing a true effect; this is the usual, somewhat "forgiving" case. When sensitivity or specificity differs by exposure group (differential misclassification — e.g., exposed patients are surveilled more intensively, raising detection), the bias can go in either direction and can manufacture or mask an association entirely. Establishing that an outcome algorithm has the same operating characteristics across arms — or measuring how they differ — is therefore a central validity task, not a footnote.

Pros, cons, and trade-offs

- vs PPV/NPV (`ppv-npv-rwe`): Sensitivity/specificity are intrinsic to the test and transport across populations of differing prevalence; they are what a validation study estimates. Their cost is that they do not tell a downstream analyst the probability that an algorithm-positive patient is truly a case — that is PPV, which depends on prevalence. Prefer sensitivity/specificity to characterize and compare algorithms and to feed misclassification corrections; prefer PPV/NPV to interpret what a flagged record means in a specific cohort. - vs likelihood ratios (`likelihood-ratios-diagnostic-rwe`): LR+ = sensitivity/(1 − specificity) and LR− = (1 − sensitivity)/specificity repackage the same two numbers into a single prevalence-independent quantity that updates pre-test odds to post-test odds, which is more directly usable at the bedside. Sensitivity/specificity remain the more transparent reporting pair and the inputs to LRs. They are complements, not substitutes. - vs ROC/AUC (`roc-auc-discrimination-rwe`): Sensitivity and specificity are evaluated at a single chosen threshold; the ROC curve plots sensitivity against (1 − specificity) across all thresholds and the AUC summarizes threshold-free discrimination. Prefer a single sensitivity/specificity pair when the algorithm has a fixed operational cut (most claims phenotypes are binary), prefer ROC/AUC when comparing continuous scores or choosing a threshold.

When to use

Report sensitivity and specificity (each with an exact binomial 95% CI) whenever you validate a computable phenotype or a claims/EHR outcome algorithm against a reference standard and you have sampled — or can reconstruct — both true-positive and true-negative strata; when you need operating characteristics that transport across cohorts of different prevalence; and when you must feed a non-differential or differential misclassification correction (e.g., the matrix/Rogan-Gladen adjustment) for an effect estimate. They are the right summary when the design samples on true status.

When NOT to use — and when it is actively misleading or dangerous

- When you only sampled algorithm-positives. Pulling charts only on patients the algorithm flagged yields PPV, not sensitivity: you never observe the false negatives (true cases the algorithm missed), so sensitivity is unidentifiable from that design. Reporting a "sensitivity" from an algorithm-positive-only chart review is a category error and the most common misuse in claims validation. - As a substitute for what a flagged record means. A 95%-sensitive, 95%-specific algorithm applied to a 1%-prevalence outcome still has a PPV near 16% — most flags are false. Quoting only the high sensitivity/specificity to reassure that "the algorithm is accurate" hides that, in a rare-outcome cohort, the positives are mostly wrong. - Across differing case spectra without checking transportability. Sensitivity estimated on severe inpatient-confirmed cases overstates sensitivity in a community cohort with milder disease (spectrum effect); transporting it silently biases any downstream correction. - When errors are differential and you treat them as non-differential. Assuming non-differential misclassification (and the comforting toward-the-null attenuation) when surveillance or coding intensity differs by exposure can bias the effect in either direction — including manufacturing a spurious association.

Data-source operational depth

- Claims (FFS): The "test" is a code algorithm (ICD-10/CPT/NDC with position and setting rules); the reference is adjudicated chart review or registry linkage on a sample drawn from both algorithm-positive and algorithm-negative strata so that FN are observable. Medicare Advantage enrollees generate no fee-for-service claims, so MA-only person- time produces apparent algorithm-negatives that are merely unobserved — restrict the validation frame to FFS-observable person-time or sensitivity is biased upward. Sensitivity is expensive to estimate well because true cases the algorithm misses are, by construction, hard to find without a high-coverage reference. - EHR: Structured problem lists, labs, and NLP on notes sharpen the algorithm, but encounter-driven capture means a patient seen elsewhere ("leakage") looks negative though truly positive, depressing apparent sensitivity differentially by how tied each patient is to the system. Define an "active in system" requirement before estimating operating characteristics. - Registry: Often serves as the reference standard (adjudicated incident events). Use registry linkage to enumerate true positives the claims/EHR algorithm missed, which is the only credible route to a defensible sensitivity.

Worked claims example

A team validates a claims algorithm for incident heart failure ("1 inpatient principal-dx HF claim OR 2 outpatient HF claims >=30 days apart within 365 days") against adjudicated chart review in a commercial + Medicare FFS sample, restricting to FFS-observable person-time. They draw a stratified validation sample: 300 algorithm- positive and 300 algorithm-negative charts, then adjudicate true status. Among the 300 algorithm-positives, 261 are confirmed HF (TP = 261, FP = 39); among the 300 algorithm-negatives, 18 turn out to be true HF the algorithm missed (FN = 18, TN = 282). Sensitivity = 261 / (261 + 18) = 0.935 (93.5%) and specificity = 282 / (282 + 39) = 0.879 (87.9%) — note these are estimated within strata and the design (sampling on the algorithm result, then re-weighting to the source prevalence) is what makes sensitivity identifiable here because algorithm-negatives were charted. Report exact binomial 95% CIs (sensitivity 0.90–0.96; specificity 0.84–0.91). Because incident HF detection is plausibly more intensive in one treatment arm (closer follow-up), the team checks whether sensitivity differs by exposure before assuming non-differential attenuation, and carries the (sensitivity, specificity) pair into a matrix misclassification correction of the downstream risk ratio.

Interpreting the output

In the worked example, chart-review validation of a heart-failure algorithm yields sensitivity = 0.935 and specificity = 0.879 (exact binomial 95% CIs: 0.90–0.96 and 0.84–0.91).

(1) Formal interpretation. Sensitivity of 0.935 means that, among patients confirmed by adjudicated chart review to have incident heart failure, 93.5% are correctly captured by the claims algorithm; 6.5% are missed (false-negative rate). Specificity of 0.879 means that, among confirmed non-cases, 87.9% are correctly excluded; 12.1% are incorrectly flagged (false-positive rate). Both estimates are conditioned on known true status — they are properties of the algorithm in this case-spectrum and data-source context, not of the underlying disease frequency in the broader population. Prevalence does not appear in either formula. The 95% CIs quantify sampling uncertainty around the estimated proportions given the validation sample size.

(2) Practical interpretation. High sensitivity (0.935) indicates the algorithm captures most true cases, limiting outcome under-ascertainment. The imperfect specificity (0.879) means some algorithm- positives are not true cases — a source of non-differential misclassification that attenuates the downstream risk ratio toward the null in a predictable, quantifiable direction. Before assuming non-differential attenuation, check whether sensitivity differs by exposure arm (differential misclassification). Carry the (sensitivity, specificity) pair into a matrix misclassification-bias correction to recover a less-biased effect estimate. These estimates may not transport to populations with a different disease spectrum or coding environment.

Worked example

Scenario

A team built a claims-code rule (an algorithm) to flag patients with heart failure, and they want to know how good it is. They take 300 patients whose true heart-failure status is already known from adjudicated chart review: 100 truly have heart failure and 200 truly do not. They run the algorithm on all 300 and sort everyone into a 2x2 confusion table, then compute sensitivity and specificity.

Dataset

The 2x2 confusion table an analyst fills in: columns are the truth from chart review, rows are what the algorithm said. Each cell is a count of patients.

Steps

• Read the truth columns: 100 patients truly have heart failure (80 TP + 20 FN) and 200 truly do not (20 FP + 180 TN).

• Sensitivity asks: of the truly-diseased people, what share did the algorithm correctly flag? Go down the Disease + column: TP / (TP + FN) = 80 / (80 + 20) = 80 / 100.

• Specificity asks: of the truly-disease-free people, what share did the algorithm correctly clear? Go down the Disease - column: TN / (TN + FP) = 180 / (180 + 20) = 180 / 200.

• Each calculation stays inside one truth column, which is why the answers don't change if the disease becomes more or less common.

Result

Sensitivity = 80 / (80 + 20) = 80 / 100 = 0.80 (80%): the algorithm catches 80% of true heart-failure patients and misses 20%. Specificity = 180 / (180 + 20) = 180 / 200 = 0.90 (90%): the algorithm correctly clears 90% of patients without heart failure and false-alarms on 10%.

Runnable example

from scipy.stats import beta

def exact_binom_ci(k: int, n: int, conf: float = 0.95):
    # Clopper-Pearson exact interval for a proportion k/n.
    alpha = 1.0 - conf
    lo = 0.0 if k == 0 else beta.ppf(alpha / 2, k, n - k + 1)
    hi = 1.0 if k == n else beta.ppf(1 - alpha / 2, k + 1, n - k)
    return lo, hi

def sens_spec(tp: int, fp: int, fn: int, tn: int) -> dict:
    # Sensitivity conditions on true positives (TP + FN); specificity on true negatives (TN + FP).
    sens = tp / (tp + fn)
    spec = tn / (tn + fp)
    s_lo, s_hi = exact_binom_ci(tp, tp + fn)
    p_lo, p_hi = exact_binom_ci(tn, tn + fp)
    return {
        "sensitivity": sens, "sensitivity_ci": (s_lo, s_hi),
        "specificity": spec, "specificity_ci": (p_lo, p_hi),
    }

# Worked claims example: HF algorithm validated by stratified chart review.
res = sens_spec(tp=261, fp=39, fn=18, tn=282)
print(f"Sensitivity = {res['sensitivity']:.3f}  95% CI {res['sensitivity_ci'][0]:.3f}-{res['sensitivity_ci'][1]:.3f}")
print(f"Specificity = {res['specificity']:.3f}  95% CI {res['specificity_ci'][0]:.3f}-{res['specificity_ci'][1]:.3f}")

library(epiR)

# Worked HF-algorithm example: rows = algorithm (+/-), columns = truth (D+/D-).
#            D+    D-
# algo +    261    39      (TP, FP)
# algo -     18   282      (FN, TN)
tab <- as.table(matrix(c(261, 39, 18, 282), nrow = 2, byrow = TRUE,
                       dimnames = list(Algorithm = c("pos", "neg"),
                                       Truth     = c("Dpos", "Dneg"))))

res <- epi.tests(tab, conf.level = 0.95)
print(res)                 # sensitivity, specificity (with exact 95% CIs), plus PPV/NPV/LRs
res$detail[res$detail$statistic %in% c("se", "sp"), ]   # isolate sensitivity & specificity rows