F1 Score, Precision, and Recall

In plain language

When a computer algorithm scans health records to flag patients who have a particular disease, it will sometimes flag a healthy patient by mistake (a false positive) and sometimes miss a true patient (a false negative). Precision answers 'of everyone the algorithm flagged, what fraction actually had the disease?' and recall answers 'of everyone who truly had the disease, what fraction did the algorithm catch?' The F1 score blends these two rates into a single number by taking their harmonic mean — a formula that gives a low score whenever either rate is poor, so you can't game it by doing well on only one side. It runs from 0 (useless) to 1 (perfect), and is the preferred single-number summary when the outcome is rare, like most diseases in a large insurance claims database.

Precision

, recall, and the F1 score summarize a binary classifier (or a computable phenotype / outcome algorithm) entirely from its 2x2 confusion matrix at a chosen decision threshold. Let TP, FP, FN, TN be the true positives, false positives, false negatives, and true negatives. Then precision = TP / (TP + FP) — identical to the positive predictive value (PPV), the probability that a flagged record truly has the outcome — and recall = TP / (TP + FN) — identical to sensitivity, the probability that a true case is flagged. The F1 score is the harmonic mean of the two, F1 = 2 precision recall / (precision + recall) = 2TP / (2TP + FP + FN). The harmonic mean (rather than the arithmetic mean) is used because it punishes imbalance between precision and recall: a classifier with precision 0.95 and recall 0.05 has arithmetic mean 0.50 but F1 = 0.095, correctly reflecting that it is nearly useless. The general F-beta score, F_beta = (1 + beta^2) precision recall / (beta^2 * precision + recall), weights recall beta times as heavily as precision; F2 (beta=2) favors recall (miss few true cases), F0.5 (beta=0.5) favors precision (few false alarms).

Core conceptual distinction

Precision, recall, and F1 are all threshold-specific point estimates — they describe one operating point, not a classifier's whole ranking ability. To characterize behavior across thresholds you trace a precision-recall (PR) curve (precision on the y-axis against recall on the x-axis as the decision threshold sweeps), the natural analogue of the ROC curve. The single most important property for RWE: none of precision, F1, or the PR curve uses the true-negative count (TN), whereas the ROC curve's specificity = TN/(TN+FP) and the false-positive rate do. When the outcome is rare — the norm for adverse events, incident cancers, or phenotypes in claims/EHR, where prevalence may be <1% — TN dominates the denominator of specificity, so an enormous absolute number of false positives barely moves the false-positive rate and the ROC curve (and its AUC) looks deceptively excellent. The PR curve, by ignoring TN, exposes that the same classifier may flag ten false positives for every true case (precision 0.09). Saito & Rehmsmeier (2015) and Davis & Goadrich (2006) formalize this: a curve that dominates in ROC space also dominates in PR space, but PR space visually resolves differences in the high-imbalance regime that ROC space compresses against the y-axis.

Pros, cons, and trade-offs

- vs ROC / AUC (`roc-auc-discrimination-rwe`): Precision/recall/F1 and the PR curve are threshold- and prevalence-aware — precision answers "of the records this algorithm flags, what fraction are real?", which is exactly the chart-review and downstream-analytic burden a pharmacoepi team cares about. ROC/AUC are prevalence-invariant (they depend only on the score ranking, not on case mix), which makes AUC portable across populations but blind to the false-positive flood under rare outcomes. Prefer PR/F1 when the positive class is rare and the cost of false positives is real (manual adjudication, biased downstream rate estimates); prefer ROC/AUC when comparing intrinsic discrimination across cohorts with different prevalence, or when both error types are roughly symmetric. - vs raw accuracy: Accuracy = (TP+TN)/N is actively misleading under imbalance — a classifier that calls every record "no outcome" achieves 99% accuracy when prevalence is 1%, while having recall 0 and undefined precision. F1 cannot be gamed this way. Never report accuracy alone for a rare outcome. - vs Matthews correlation coefficient (MCC) / balanced accuracy: MCC uses all four cells and is symmetric in the two classes, so it is more robust when you care about both positive and negative prediction (Chicco & Jurman, 2020). F1 is asymmetric — it ignores TN and treats the positive class as the focus — which is exactly right for case-finding but can flatter a classifier that is good only on the majority of cases. Prefer MCC when both classes matter symmetrically; prefer F1 when the task is finding the rare positives. - vs PPV/sensitivity reported separately (`claims-outcome-algorithm-ppv-sensitivity-rwe`): F1 collapses two numbers into one for model selection, but it hides the operating-point trade-off the validation literature insists on reporting. For a phenotype headed into an analytic cohort, report precision (PPV) and recall (sensitivity) separately with confidence intervals; use F1 only as a tie-breaker or hyperparameter-selection objective.

When to use

Tuning and selecting a computable phenotype, outcome algorithm, or ML risk classifier where the positive class is rare and false positives carry real cost (adjudication labor, attenuated downstream effect estimates from outcome misclassification). Use the PR curve and its area (average precision) to compare candidate algorithms across all thresholds; use F1 (or F-beta with a justified beta) as a single scalar objective for cross-validated model selection; use precision at a fixed recall (or recall at a fixed precision) when the deployment threshold is dictated by an operational constraint (e.g., "we can chart-review 200 flags").

When NOT to use — and when it is actively misleading or dangerous

- Comparing classifiers across populations with different prevalence. Precision and F1 change with prevalence even when the classifier is identical, because PPV depends on base rate. Comparing F1 across two databases with different outcome prevalence confounds algorithm quality with case mix; use AUC or report prevalence alongside. - As the only validation metric for a phenotype entering a causal analysis. A high F1 does not tell you whether misclassification is differential by exposure — the property that biases comparative estimates. F1 is a marginal accuracy summary, not a bias diagnostic; pair it with separate PPV/sensitivity and a quantitative-bias-analysis correction. - When the negative class genuinely matters. If a false negative and a false positive are both clinically costly and you care about correctly classifying non-cases, F1's blindness to TN understates the problem; use MCC or balanced accuracy. - Reporting a single F1 without the threshold or the PR curve. F1 at an unstated threshold is uninterpretable and cherry-picking the threshold that maximizes F1 on the same data that estimates it overstates performance; choose the threshold on training/validation folds and report the held-out F1 with its CI.

Data-source operational depth

- Claims (FFS): The "truth" labels for precision/recall almost always come from chart review on a sampled subset, so precision (PPV) is estimated directly from the reviewed flags but recall (sensitivity) requires sampling true cases from an independent reference (a registry, linked EHR, or adjudication) — you cannot estimate recall from claims alone because the FN cell is unobserved. Restrict the confusion matrix to FFS-observable person-time; Medicare Advantage enrollees generate no claims, so apparent "negatives" in MA-only spans are unobserved, not true negatives, and silently inflate precision. Outcome prevalence in claims is typically <2%, so report the PR curve, not the ROC curve. - EHR: Richer features (labs, vitals, NLP-derived concepts) usually raise both precision and recall, but encounter-driven ascertainment means a true case who seeks care out-of-system is an unobservable false negative; recall is therefore an overestimate unless an external reference standard captures out-of-system events. Define an "active in system" requirement before counting negatives. - Registry / linked: Adjudicated registry outcomes provide the cleanest reference standard for both precision and recall, making linked claims-EHR-registry the strongest substrate for honest PR estimation; linkage selects the linkable subset, so report the linkage rate and check that the validation sample's prevalence matches the analytic cohort's.

Worked example

A claims-based computable phenotype for incident heart failure is applied to 100,000 FFS-observable person-records with true prevalence ~1.5% (1,500 true cases). At the chosen threshold the algorithm flags 2,000 records, of which 1,200 are confirmed by linked-EHR adjudication (TP=1,200, FP=800), and 300 true cases are missed (FN=300). Then precision (PPV) = 1200/(1200+800) = 0.60, recall (sensitivity) = 1200/(1200+300) = 0.80, and F1 = 20.600.80 / (0.60+0.80) = 0.96/1.40 = 0.686. The ROC view flatters this algorithm: with TN ~= 98,000, the false-positive rate is only 800/98,800 = 0.008, so the ROC curve hugs the top-left corner and AUC looks near-perfect — yet two in five flagged records are false, a 40% adjudication-waste and a source of outcome misclassification. The PR curve makes that visible. If the downstream analysis cannot tolerate the false positives, slide the threshold up (raising precision, lowering recall) and report F0.5 to formalize the precision preference; if missing true cases is the dominant cost, slide down and report F2.

Interpreting the output

In the small worked example, TP = 80, FP = 20, FN = 40 yields precision = 0.800, recall = 0.667, and F1 = 0.727. In the large-scale example (TP = 1,200, FP = 800, FN = 300), F1 = 0.686.

(1) Formal interpretation. F1 is the harmonic mean of precision (= TP / (TP + FP)) and recall (= TP / (TP + FN)). The harmonic mean penalizes imbalance: a model with precision 0.99 but recall 0.01 earns an F1 near 0.02, exposing its failure despite perfect avoidance of false positives. True negatives do not enter the F1 formula — this is deliberate. When the negative class vastly outnumbers the positive class (as in rare-disease phenotyping), accuracy is dominated by the TN count and can exceed 99% even for a model that flags nothing at all; F1 and the PR curve see through that illusion because they are computed only over the positive-class predictions and ground-truth positives. An F1 of 0.686 in the large example reflects that 40% of flags are false (precision 0.60) even though recall is high (0.80).

(2) Practical interpretation. When evaluating a phenotyping algorithm for use as an RWE outcome, both error types matter but often unequally. If false positives (misclassified non-cases used as outcomes) drive attenuation bias, prioritize precision and consider threshold increases or F0.5. If false negatives (missed cases causing outcome under-ascertainment) are more damaging, prioritize recall and consider F2. Always plot the full precision-recall curve across thresholds rather than reporting a single F1 at one cut-point — the area under the PR curve (AUCPR) captures classifier performance more honestly than ROC-AUC under class imbalance.

Worked example

Scenario

A research team builds a claims-based algorithm to identify patients who developed a serious drug reaction in a 300-patient sample. A physician reviews all 300 charts and confirms the truth. The algorithm flags 100 patients as cases. Of those 100 flagged patients, 80 truly had the drug reaction (TP=80) and 20 did not (FP=20). Of the 200 patients the algorithm did not flag, 40 truly had the reaction and were missed (FN=40) and 160 were correctly cleared (TN=160). The team wants to know how well the algorithm performs.

Dataset

2x2 confusion matrix: adjudicated truth (rows) versus algorithm flag (columns) for 300 patients.

Steps

• Precision = TP / (TP + FP) = 80 / (80 + 20) = 80 / 100 = 0.800. This means 80 out of every 100 patients the algorithm flags truly had the drug reaction; 1 in 5 flags is a false alarm.

• Recall = TP / (TP + FN) = 80 / (80 + 40) = 80 / 120 = 0.667. This means the algorithm caught 80 of the 120 true cases; 1 in 3 true cases was missed.

• F1 = 2 × precision × recall / (precision + recall) = 2 × 0.800 × 0.667 / (0.800 + 0.667) = 1.067 / 1.467 = 0.727.

• Cross-check with the shortcut formula: F1 = 2×TP / (2×TP + FP + FN) = 160 / (160 + 20 + 40) = 160 / 220 = 0.727. Same answer.

• Because the harmonic mean punishes imbalance, an algorithm that had precision 1.0 but recall 0.0 would score F1 = 0, even though its regular average would be 0.5 — this is why F1 can't be fooled by doing well on only one side.

Result

Precision = 0.800 (80 of 100 flags are real cases). Recall = 0.667 (80 of 120 true cases caught). F1 = 0.727 (harmonic mean of the two). The algorithm is reasonably precise but misses one-third of true cases; whether that trade-off is acceptable depends on the cost of missing a case versus the cost of a false alarm.

Runnable example

import numpy as np
from sklearn.metrics import (precision_score, recall_score, f1_score, fbeta_score,
                             precision_recall_curve, average_precision_score,
                             confusion_matrix)

# Worked example: rare outcome (~1.5% prevalence). Reconstruct labels matching the
# confusion matrix TP=1200, FP=800, FN=300, TN=97700 at the chosen threshold.
y_true  = np.r_[np.ones(1200), np.zeros(800), np.ones(300), np.zeros(97700)].astype(int)
y_pred  = np.r_[np.ones(1200), np.ones(800),  np.zeros(300), np.zeros(97700)].astype(int)

tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
prec = precision_score(y_true, y_pred)            # = PPV  = TP/(TP+FP) = 0.60
rec  = recall_score(y_true, y_pred)               # = sens = TP/(TP+FN) = 0.80
f1   = f1_score(y_true, y_pred)                   # harmonic mean        = 0.686
f2   = fbeta_score(y_true, y_pred, beta=2)        # recall-weighted
f05  = fbeta_score(y_true, y_pred, beta=0.5)      # precision-weighted
print(f"TP={tp} FP={fp} FN={fn} TN={tn}")
print(f"precision(PPV)={prec:.3f} recall(sens)={rec:.3f} F1={f1:.3f} F2={f2:.3f} F0.5={f05:.3f}")

# PR curve from continuous scores: average precision is the imbalance-aware summary.
# The no-skill baseline equals prevalence (here ~0.015), unlike ROC's 0.5 baseline.
rng = np.random.default_rng(0)
score = np.where(y_true == 1, rng.beta(2.5, 2.0, size=y_true.size),
                              rng.beta(2.0, 6.0, size=y_true.size))
p, r, thr = precision_recall_curve(y_true, score)
ap = average_precision_score(y_true, score)
print(f"average precision (area under PR curve) = {ap:.3f}; "
      f"no-skill baseline = prevalence = {y_true.mean():.3f}")

library(yardstick)
library(PRROC)

# Worked example confusion matrix: TP=1200, FP=800, FN=300, TN=97700.
truth <- factor(c(rep("yes",1200), rep("no",800), rep("yes",300), rep("no",97700)),
                levels = c("yes","no"))
pred  <- factor(c(rep("yes",1200), rep("yes",800), rep("no",300), rep("no",97700)),
                levels = c("yes","no"))
df <- data.frame(truth = truth, pred = pred)

# event_level='first' so "yes" (the rare positive) is the event of interest.
precision(df, truth, pred, event_level = "first")   # PPV  = 0.60
recall(df,    truth, pred, event_level = "first")   # sens = 0.80
f_meas(df,    truth, pred, event_level = "first")   # F1   = 0.686
f_meas(df,    truth, pred, event_level = "first", beta = 2)    # F2  (recall-weighted)
f_meas(df,    truth, pred, event_level = "first", beta = 0.5)  # F0.5 (precision-weighted)

# PR curve area from continuous scores; baseline precision = prevalence, not 0.5.
y    <- ifelse(truth == "yes", 1L, 0L)
set.seed(0)
score <- ifelse(y == 1, rbeta(length(y), 2.5, 2.0), rbeta(length(y), 2.0, 6.0))
pr <- pr.curve(scores.class0 = score[y == 1], scores.class1 = score[y == 0], curve = TRUE)
cat("area under PR curve (average precision) =", round(pr$auc.integral, 3), "\n")