ROC Curves, AUC, and the c-statistic

In plain language

A risk score (like a model that outputs a number from 0 to 1 for each patient) is only useful if it actually separates sick patients from healthy ones — giving higher numbers to the people who will get the disease and lower numbers to those who won't. The AUC (Area Under the Curve) measures exactly that ability: it asks, if you picked one patient who got the disease and one who didn't, how often does the sick patient have the higher score? An AUC of 0.5 means the score is no better than a coin flip, and 1.0 means it perfectly ranks every sick patient above every healthy one. The AUC only measures ranking order, not whether the predicted numbers are accurate — a score could be a perfect ranker while still predicting wildly wrong probabilities.

The receiver operating characteristic (ROC) curve displays the full trade-off between sensitivity and specificity as the decision threshold of a continuous (or ordinal) score sweeps from most to least stringent. For every candidate threshold the curve plots the true-positive rate (sensitivity) on the y-axis against the false-positive rate (1 - specificity) on the x-axis; the resulting monotone curve runs from (0,0) to (1,1). A useless score lies on the diagonal (sensitivity = 1 - specificity at every cut); a better score bows toward the top-left corner. The area under the ROC curve (AUC) collapses the whole curve into one number in [0.5, 1] (0.5 = no discrimination, 1 = perfect).

Core idea — AUC as concordance

The AUC has a clean probabilistic meaning that does not mention thresholds at all:

AUC = P(score of a random case > score of a random non-case),

i.e. the probability that the model ranks a randomly chosen true positive above a randomly chosen true negative (ties split). This makes the AUC a pure measure of discrimination — how well the score orders cases above non-cases — and it is numerically identical to the Mann-Whitney U statistic divided by the product of the two group sizes, and to Harrell's c-statistic (concordance index) for binary outcomes. For survival outcomes the c-statistic generalizes to the probability that, among comparable (orderable) pairs, the patient who fails earlier had the higher predicted risk. Because it depends only on the ordering of scores, the AUC is threshold-free and invariant to any monotone transformation of the score.

Discrimination is not calibration

This is the central conceptual distinction and the most consequential one in RWE. Discrimination asks whether the model separates cases from non-cases (ranking); calibration asks whether the predicted probabilities match observed event frequencies (e.g., among patients predicted 20%, do ~20% have the event?). A model can have excellent AUC and be badly miscalibrated — systematically predicting risks twice too high — because any strictly increasing recalibration of the scores leaves the ordering, and therefore the AUC, completely unchanged. For decision-making and HTA, calibration and net benefit often matter more than AUC; the AUC alone can endorse a model that would mislead at every clinically used threshold. Report the AUC with a calibration assessment (calibration plot, calibration-in-the-large and slope), never as a stand-alone verdict.

Limitations beyond calibration

(1) Class imbalance / rare outcomes: the AUC is computed over all case/non-case pairs and is largely insensitive to prevalence, which is a virtue for comparability but a vice for decision relevance — a high AUC in a 1%-event setting can still correspond to poor positive predictive value and little clinical usefulness; the precision-recall curve and decision-curve (net-benefit) analysis are more informative when positives are rare. (2) Insensitivity to clinically meaningful changes: adding a strong new predictor often barely moves the AUC even when it improves risk stratification at decision-relevant thresholds, which is why reclassification and net-benefit measures supplement it. (3) Overoptimism: an AUC computed on the same data used to fit the model is upward-biased; report an internally validated (bootstrap- or cross-validation-corrected) and, ideally, externally validated AUC.

Pros, cons, and trade-offs

- vs sensitivity/specificity at a fixed threshold (`sensitivity-specificity-rwe`): The AUC summarizes performance across all thresholds in one number and is the right tool when comparing continuous scores or when no threshold is yet fixed; a single sensitivity/specificity pair is one point on the ROC and is what matters once an operational cut is chosen. Prefer AUC for model comparison and threshold selection; prefer the operating-point pair for a deployed binary rule. - vs calibration / net benefit (`prediction-model-validation-recalibration-rwe`): The AUC is invariant to monotone recalibration and so says nothing about whether predicted probabilities are right; calibration and decision-curve analysis directly address clinical usefulness. Prefer the AUC as a discrimination headline; always pair it with calibration and net benefit before claiming a model is fit for decisions. - vs precision-recall / PPV at a threshold (`ppv-npv-rwe`): Under heavy class imbalance the ROC can look strong while PPV is poor; the PR curve and PPV expose the rare-positive reality the AUC obscures. Prefer the AUC for a prevalence-stable discrimination summary; prefer PR/PPV when the practical question is "of those flagged, how many are real?"

When to use

Report the AUC (with a 95% CI) to summarize how well a continuous risk score or diagnostic marker discriminates cases from non-cases; to compare competing models on a common scale independent of any threshold; as the headline discrimination metric in a prediction-model development/validation paper (alongside calibration per TRIPOD); and to select or justify a clinical decision threshold by inspecting the ROC's sensitivity/specificity trade-off. Use the DeLong method (or bootstrap) to compare two AUCs on the same patients.

When NOT to use — and when it is actively misleading or dangerous

- As the sole evidence a model is good. Quoting a high AUC while omitting calibration can endorse a model that systematically over- or under-predicts risk and would harm decisions at every used threshold — the most dangerous and common misuse. AUC measures ranking, not the correctness of probabilities. - As a clinical-utility verdict under class imbalance. In a rare-outcome setting a strong AUC can coexist with a PPV that makes the model useless for action; presenting AUC alone hides this. Add PR/decision-curve analysis. - For detecting the value of a new predictor. The AUC is notoriously insensitive to adding a genuinely useful marker; concluding "the new variable adds nothing" because the AUC barely moved is misleading — use reclassification/net benefit. - On the training data without correction. An apparent (resubstitution) AUC overstates performance; reporting it as if it were validated misrepresents how the model will discriminate in new patients. - Comparing AUCs across studies with different case spectra. Because discrimination depends on the spread of risk in the sample, an AUC from a heterogeneous (wide-spectrum) cohort is not comparable to one from a narrow-spectrum cohort; the difference can be spectrum, not model quality.

Data-source operational depth

- Claims (FFS): The score is typically a fitted risk model (logistic for a binary outcome, Cox for time-to-event) over claims-derived predictors (comorbidity indices, prior utilization, NDC classes); the AUC/c-statistic measures how well it ranks who will have the outcome. Restrict to FFS-observable person-time so the outcome label is not corrupted by Medicare Advantage spans (no fee-for-service claims), which would misclassify events and inflate or deflate apparent discrimination. Report an internally validated (bootstrap-corrected) AUC because claims models are often fit and evaluated on the same extract. - EHR: Richer continuous predictors (labs, vitals, NLP features) usually improve discrimination, but encounter-driven capture means outcome ascertainment leakage mislabels some non-cases, biasing the AUC; for time-to-event use the time-dependent / inverse-probability-of-censoring-weighted c-statistic to handle censoring honestly. - Registry/linked: Registry-adjudicated outcomes give the cleanest label for the case/non-case ranking; linked claims- EHR-registry data support external validation, where the AUC's transportability (and any spectrum shift) is the key question.

Worked claims example

A team develops a logistic model for 1-year incident heart failure in a commercial + Medicare FFS cohort (FFS-observable person-time only), using age, sex, a comorbidity index, prior cardiac utilization, and baseline NDC classes; the linear predictor is the continuous score. On a held-out validation extract of 4,000 patients (320 events, 8% prevalence) they sweep the threshold to trace the ROC and obtain AUC = 0.781 (95% CI 0.756-0.806) by the DeLong method — equivalently, a randomly chosen future HF case has a higher predicted risk than a randomly chosen non-case about 78% of the time. They do not stop there: the calibration plot shows the model over-predicts in the top decile (predicted 30% vs observed 22%), so despite acceptable discrimination it needs recalibration before use; and because the outcome is uncommon, they add a decision-curve analysis showing positive net benefit only over a narrow threshold range. A rival model adds an EHR-derived ejection-fraction feature and raises the AUC to only 0.792 — a difference whose DeLong test p = 0.07 and whose practical value is better judged by net reclassification at the decision threshold than by the small AUC change. The headline AUC is reported with its CI, the internal-validation optimism correction, and — mandatorily — the calibration assessment alongside.

Interpreting the output

In the clinical example, the risk-score model achieves an AUC (c-statistic) of 0.781. In the small worked example with 8 concordant of 9 discordant pairs, AUC ≈ 0.889.

(1) Formal interpretation. AUC = 0.781 means that, if one true case and one true non-case are drawn at random from the population, the model assigns the higher predicted risk to the case approximately 78.1% of the time. This is a rank-order (concordance) statement — it describes the model’s ability to separate cases from non-cases across all possible thresholds simultaneously. It says nothing about whether the predicted probabilities are numerically accurate (calibration). AUC is threshold-free: no single decision cut-point is assumed, making it a summary of the full ROC curve. A model could have high AUC but poor calibration if it consistently ranks cases above non-cases while predicting systematically inflated or deflated absolute probabilities.

(2) Practical interpretation. AUC = 0.781 reflects good but not excellent discrimination — the model ranks cases above non-cases about four times in five. As the worked example demonstrates, AUC alone is insufficient: if the calibration plot reveals systematic over-prediction in the top risk decile, recalibration is needed before clinical or administrative use. Report AUC with its CI, any internal-validation optimism correction, and a calibration assessment (calibration plot, Brier score, or E/O ratio). Do not interpret AUC as accuracy or use it as the sole criterion for model selection — evaluate net reclassification and decision-curve analysis at the decision threshold that matters.

Worked example

Scenario

A research team built a logistic regression model to predict which patients will be hospitalized for heart failure within one year. The model outputs a predicted risk score between 0 and 1 for every patient. They want to know how well the model discriminates — does it actually rank future hospitalizations above non-hospitalizations? They test it on six patients: three who were later hospitalized (cases) and three who were not (non-cases).

Dataset

Six patients with their true one-year hospitalization outcome and the model's predicted risk score.

Steps

• Identify the cases (true_outcome = 1): PT-101 (0.90), PT-102 (0.85), PT-103 (0.60). Identify the non-cases (true_outcome = 0): PT-104 (0.80), PT-105 (0.40), PT-106 (0.20).

• The AUC equals the probability that a randomly chosen case scores higher than a randomly chosen non-case. To compute it by hand, list every possible case/non-case pair — with 3 cases and 3 non-cases there are 3 × 3 = 9 pairs total.

• Check each pair: PT-101 (0.90) vs PT-104 (0.80) → case wins (concordant). PT-101 (0.90) vs PT-105 (0.40) → case wins (concordant). PT-101 (0.90) vs PT-106 (0.20) → case wins (concordant).

• Continue: PT-102 (0.85) vs PT-104 (0.80) → case wins (concordant). PT-102 (0.85) vs PT-105 (0.40) → case wins (concordant). PT-102 (0.85) vs PT-106 (0.20) → case wins (concordant).

• Continue: PT-103 (0.60) vs PT-104 (0.80) → non-case wins (discordant). PT-103 (0.60) vs PT-105 (0.40) → case wins (concordant). PT-103 (0.60) vs PT-106 (0.20) → case wins (concordant).

• Count the results: 8 concordant pairs, 1 discordant pair, 0 ties. AUC = concordant pairs / total pairs = 8 / 9 ≈ 0.889.

• Interpretation: In about 89% of all case/non-case matchups, the model correctly ranks the future hospitalization patient higher. Note that PT-103 (a true case with a moderate score of 0.60) was outranked by PT-104 (a true non-case with a score of 0.80) — the one failure of the model on this small dataset.

Result

AUC = 8 concordant pairs / 9 total pairs = 0.889. The model correctly ranks a randomly chosen future hospitalization above a randomly chosen non-hospitalization about 89% of the time. This is solid discrimination — but it says nothing about whether the score of 0.90 or 0.60 is an accurate probability; that would require a separate calibration check.

Runnable example

import numpy as np
from sklearn.metrics import roc_auc_score, roc_curve
from scipy.stats import mannwhitneyu, norm

# Illustrative: predicted risk scores and true binary HF outcomes (1 = event).
rng = np.random.default_rng(7)
y = np.r_[np.ones(40), np.zeros(160)]                 # 20% prevalence
score = np.r_[rng.normal(0.7, 0.25, 40),              # cases score higher on average
              rng.normal(0.4, 0.25, 160)]

auc = roc_auc_score(y, score)                         # = c-statistic
fpr, tpr, thr = roc_curve(y, score)                   # 1-specificity, sensitivity across thresholds

# AUC as concordance: Mann-Whitney U / (n_case * n_ctrl) gives the same number.
U, _ = mannwhitneyu(score[y == 1], score[y == 0], alternative="greater")
auc_mw = U / (np.sum(y == 1) * np.sum(y == 0))

# Hanley-McNeil (1982) standard error for a single AUC -> normal-approx 95% CI.
n1, n2 = int((y == 1).sum()), int((y == 0).sum())
q1 = auc / (2 - auc); q2 = 2 * auc**2 / (1 + auc)
se = np.sqrt((auc*(1-auc) + (n1-1)*(q1-auc**2) + (n2-1)*(q2-auc**2)) / (n1*n2))
z = norm.ppf(0.975)
print(f"AUC = {auc:.3f} (concordance check {auc_mw:.3f}); 95% CI {auc-z*se:.3f}-{auc+z*se:.3f}")

library(pROC)

set.seed(7)
y     <- c(rep(1, 40), rep(0, 160))                   # 20% prevalence
score <- c(rnorm(40, 0.7, 0.25), rnorm(160, 0.4, 0.25))

roc_obj <- roc(response = y, predictor = score, levels = c(0, 1), direction = "<")
auc(roc_obj)                                          # AUC = c-statistic
ci.auc(roc_obj, method = "delong")                    # DeLong 95% CI

# Compare two models on the SAME patients with the DeLong test.
score2  <- score + rnorm(200, 0, 0.10)                # rival score
roc2    <- roc(y, score2, direction = "<")
roc.test(roc_obj, roc2, method = "delong")            # paired AUC comparison