Endpoint Adjudication and Chart Review

In plain language

Endpoint adjudication is the process of having trained clinicians read a patient's actual medical records and decide whether an event the computer algorithm flagged — for example, a hospitalization coded as a heart attack — truly meets the study's definition of that event. The clinicians review source documents (discharge summaries, lab results, ECGs) against a pre-written case definition, and they do so without knowing which drug the patient received, so their judgment cannot be swayed by that knowledge. The result is a confirmed event count that is more accurate than the raw algorithm output, along with a measure of how often the two reviewers agreed — which tells you how reliable the confirmation process itself is.

Endpoint adjudication and chart review

is the operational step that converts an algorithm-identified candidate event (e.g., an inpatient claim with a primary-position ICD-10 acute MI code) into a confirmed endpoint by having trained clinical reviewers compare the underlying source documents — discharge summaries, cath/ECG/troponin results, imaging, progress notes, death certificates — against a written, pre-specified case definition (Universal Definition of MI, NINDS/Sentinel stroke criteria, KDIGO AKI staging, Bradford-Hill-style causality for adverse events). The product is not a single number but a measurement-error model: positive predictive value (PPV) of the algorithm, sensitivity/specificity when algorithm-negative records are also reviewed, and inter-rater reliability (Cohen's or weighted kappa). Those parameters are then propagated forward to correct the crude event rate and to quantify how much residual outcome misclassification could move the effect estimate.

Core conceptual distinction

. Three things are being separated and must not be conflated. (1) Algorithm identification vs adjudicated truth: the claims/EHR algorithm is a screening test; adjudication is the reference standard. The deliverable is the operating characteristics of the algorithm against that standard, not a re-counted event total. (2) Validation vs correction: estimating PPV/sensitivity (validation) is distinct from using those estimates to recover an unbiased incidence or hazard (bias correction / quantitative bias analysis). A study that reports "PPV = 0.84" but then analyzes the uncorrected algorithm-positive events as if they were all true has done validation and ignored correction. (3) Adjudication vs phenotyping: a phenotype algorithm assigns a label deterministically from structured data; adjudication is a human reference-standard process layered on top of (a sample of) those labels. The estimand the adjudication serves is whatever the parent study targets (cause-specific incidence, an as-treated hazard ratio, a decision-model transition probability) — adjudication does not change the estimand, it de-biases the outcome variable that feeds it. The cardinal design requirement is that reviewers be blinded to exposure/arm, because differential misclassification by exposure is the one error that adjudication is supposed to remove and unblinded adjudication can manufacture.

Pros, cons, and trade-offs

. - vs accepting the raw algorithm (no adjudication): Adjudication gives a defensible, quantified outcome definition and is the only way to detect and bound differential outcome misclassification, which biases relative effects in unpredictable directions. Cost: chart retrieval is expensive and slow, requires data-use agreements that many claims vendors cannot grant, and shrinks the analyzable sample to the linkable/chart- available subset. Prefer adjudication for any regulatory- or HTA-grade safety/effectiveness endpoint and for any outcome whose algorithm PPV is unknown or known to be modest (<~0.80). - vs full chart review of every event: Sampling (all positives, or a stratified sample) buys feasibility at the cost of sampling variance in PPV/sensitivity; Wilson or exact CIs and an explicit sampling frame are mandatory. Prefer a stratified probability sample when event counts are large; reserve census review for rare/serious endpoints. - vs single-reviewer abstraction: A two-independent-reviewer-plus-tiebreaker committee with a written charter produces a reportable kappa and removes idiosyncratic single-reader bias. Cost: roughly double the abstraction effort. Prefer the committee whenever the case definition involves judgment (MI subtype, stroke vs TIA, drug causality); single-reader review is acceptable only for near-mechanical confirmations. - vs probabilistic/QBA-only correction with no chart review: If charts are simply unavailable, external validation parameters from the literature can feed a quantitative bias analysis, but borrowed PPV/sensitivity from a different population/database is itself an assumption (transportability) and is weaker than in-study adjudication. Prefer in-study adjudication; fall back to QBA with sensitivity ranges only when retrieval is impossible.

When to use

. A primary or key secondary safety/effectiveness endpoint identified from claims or EHR whose algorithm operating characteristics are unknown or modest; FDA/EMA submissions and HTA dossiers where outcome validity must be demonstrated; any comparison where differential outcome capture by exposure is plausible (e.g., more surveillance in one arm); composite endpoints where component-level confirmation changes the count.

When NOT to use — and when it is actively misleading or dangerous

- Unblinded adjudication. If reviewers can see the exposure/arm, adjudication can create the differential misclassification it was meant to remove — the single most dangerous failure. Blind the packet; log and audit blinding breaks. - Adjudicating only algorithm-positives and calling the algorithm "validated." Reviewing only positives gives PPV but says nothing about sensitivity; you cannot bias-correct incidence or rule out missed events. Claiming a validated algorithm from positives alone is misleading — you must sample algorithm-negatives (the expensive step everyone skips) to estimate sensitivity. - Treating adjudicated cases as error-free ground truth. Propagating a point PPV without its CI, and without the kappa-quantified reviewer disagreement, understates uncertainty in the corrected rate. - Transporting one site's/database's PPV as a correction factor in a different population. PPV depends on prevalence and coding practice; a PPV from a tertiary cardiology cohort will overstate confirmation in a general claims population. - Cherry-picking borderline cases or letting reviewers re-adjudicate after seeing the analysis. Both break the pre-specification that makes the metric interpretable.

Data-source operational depth

. - Administrative claims (FFS): Candidate events come from diagnosis/procedure codes with position and setting (primary-position inpatient ICD-10 I21.x for acute MI). Failure modes: claims contain no clinical detail, so adjudication requires linkage to charts/EHR, which depends on a data-use agreement the vendor often does not hold — retrieval is the binding constraint, not analysis. Outpatient/ED-coded events frequently lack a discharge summary to confirm. Claims lag and adjustment mean the event packet must be assembled after run-out. Deceased patients' charts are often unobtainable, biasing the adjudicable sample away from fatal events. Medicare Advantage (MA) person-time has no FFS claims, so MA enrollees are invisible to the algorithm and unavailable for adjudication — restrict person-time to FFS (Parts A/B) or treat MA spans as unobserved, never as event-free. - EHR: Richer source documents, but external-care leakage means the adjudicator sees only the events that happened inside the network; an MI treated at a competing hospital is both unflagged and unadjudicable, deflating sensitivity differentially by how "loyal" patients are. Encounter-driven capture means structured fields may be empty even when the note documents the event — abstract from notes, and define the observation window explicitly. - Registry: Often supplies adjudicated outcomes already, but registry adjudication criteria may differ from your protocol's case definition (e.g., a registry's "stroke" includes TIA); re-adjudicate or document the criteria mismatch (a transportability problem), don't blindly inherit the registry label. - Linked claims–EHR–vital records: The strongest substrate — claims completeness + EHR clinical detail + a death index to confirm fatal endpoints — but linkage selects the linkable subset and introduces date discrepancies (claim service date vs note date vs death date) that must be reconciled before the event date is fixed.

Worked claims example

Endpoint: hospitalized acute MI in a commercial + Medicare FFS database. (1) Candidate identification: an inpatient claim with primary-position ICD-10 `I21.x` and length of stay >=1 day flags a candidate; require continuous medical enrollment in the 365 days before the index and exclude MA-only person-time so the algorithm can actually see hospitalizations. (2) De-duplication: collapse transfers and same-episode readmissions within a 30-day acute window to one event (see acute-event de-duplication). (3) Sampling: 5,000 algorithm-positive candidate MIs; draw a stratified random sample of 500 (stratified by age, sex, and fatal/non-fatal) for chart pull, plus a 250-record sample of algorithm-negative hospitalizations to estimate sensitivity. (4) Packet + charter: assemble discharge summary, serial troponins, ECG, and any catheterization report into a packet stripped of any exposure/drug information; two cardiologists independently classify each against the Fourth Universal Definition of MI; a third adjudicator breaks ties. (5) Metrics: of 500 positives, 420 confirmed -> PPV = 420/500 = 0.84 (Wilson 95% CI 0.81–0.87); inter-rater Cohen's kappa = 0.78 before tiebreak; of 250 negatives, 12 were true MIs missed by the algorithm, giving an estimated sensitivity ~ true_positives / (true_positives + estimated false negatives). (6) Bias correction: the crude algorithm rate of 5,000 events over the FFS person-time is multiplied/adjusted by the sampling-weighted PPV and sensitivity to recover the corrected incidence; (7) QBA: vary PPV across its CI and sensitivity across plausible bounds (and, for the comparative analysis, vary PPV/sensitivity separately by arm) to map how much residual outcome misclassification would change the hazard ratio — if a credible differential-misclassification scenario flips the conclusion, the unadjudicated estimate is not trustworthy.

Interpreting the output

From the full-sample worked example: 500 algorithm-positive candidates reviewed; 420 confirmed as true MI. PPV = 420 / 500 = 0.84 (Wilson 95% CI 0.81–0.87). Inter-rater Cohen's kappa = 0.78 before tiebreak. Of 250 algorithm-negative records sampled, 12 were adjudicated true MIs (false negatives).

(1) Formal interpretation. PPV = 0.84 means approximately 16% of algorithm-flagged events are false positives; the crude algorithm-positive event count overstates confirmed MIs by ≈ 19%. Kappa = 0.78 indicates substantial pre-tiebreak inter-rater agreement (Landis-Koch scale: substantial, approaching almost-perfect). The Wilson CI 0.81–0.87 reflects sampling uncertainty in the PPV estimate itself; the true adjudication PPV in the full cohort is unknown and assumed to match the validation sample's under the transportability assumption (same site mix, chart availability, and reviewer training). Under non-differential misclassification (PPV equal across arms), unconfirmed events dilute the comparative HR toward 1.0; under differential misclassification (PPV differs by arm), bias direction is unpredictable and arm-specific adjudication subsamples are required to identify it.

(2) Practical interpretation. A study reporting 5,000 algorithm-positive MIs over cohort follow-up should correct the event count to ≈ 4,200 confirmed MIs (5,000 × 0.84) for rate and cost calculations. For the comparative HR, the key question is whether PPV = 0.84 holds equally in both treatment arms; if the more-intensively monitored arm has higher PPV (say, 0.88 vs 0.80), the hazard ratio is biased away from the null — an operationally realistic concern in open-label observational settings.

Worked example

Scenario

A claims-based study of heart attack risk has a computer algorithm that scans hospital claims and flags any stay with a primary-position ICD-10 code of I21.x (acute MI) as a candidate heart attack. The study team pulls the medical records for 10 candidate events and sends each record, stripped of any drug information, to two cardiologists who independently classify each as confirmed or not confirmed. We want to measure how well the two reviewers agree and summarize that agreement with percent agreement and Cohen's kappa.

Dataset

Ten candidate MI events reviewed independently by Reviewer 1 and Reviewer 2. Y = confirmed; N = not confirmed.

Steps

• Count agreements: both reviewers said Y for events E01-E07 (7 cells); both said N for E09 and E10 (2 cells). Total agreements = 7 + 2 = 9 out of 10 events.

• Percent agreement = 9 / 10 = 0.90 (90%).

• To compute Cohen's kappa, first find each reviewer's marginal rates. Reviewer 1 said Y 8 times (E01-E07 plus E08) out of 10, so P(R1 = Y) = 8/10 = 0.80. Reviewer 2 said Y 7 times (E01-E07) out of 10, so P(R2 = Y) = 7/10 = 0.70.

• Expected agreement by chance alone: P(both Y by chance) = 0.80 x 0.70 = 0.56. P(both N by chance) = 0.20 x 0.30 = 0.06. Expected agreement P_e = 0.56 + 0.06 = 0.62.

• Kappa = (observed agreement - expected agreement) / (1 - expected agreement) = (0.90 - 0.62) / (1 - 0.62) = 0.28 / 0.38 = 0.74 (rounded to two decimal places).

• Event E08 was a disagreement (R1 confirmed, R2 did not); a third cardiologist reviewed it and confirmed it, so the final call is confirmed.

• Final confirmed event count = 8 out of 10 candidates. PPV for this small sample = 8 / 10 = 0.80. The full study (500 reviewed events, 420 confirmed) yielded PPV = 420/500 = 0.84 and kappa = 0.78 — consistent with this illustration.

Result

Percent agreement = 9/10 = 0.90. Cohen's kappa = (0.90 - 0.62) / (1 - 0.62) = 0.28 / 0.38 = 0.74. Both figures indicate substantial reviewer agreement. Eight of 10 candidate events were confirmed after tiebreak, giving a PPV of 0.80 for this small sample.

Agreement Table

Caption

2x2 table of reviewer calls before tiebreak (n = 10 candidate events)

Header Row

• R2 confirms (Y)

• R2 does not confirm (N)

• Row total

Rows

• • R1 confirms (Y)

  • 7

  • 1

  • 8

• • R1 does not confirm (N)

  • 0

  • 2

  • 2

• • Column total

  • 7

  • 3

  • 10

Runnable example

import numpy as np
import pandas as pd
from statsmodels.stats.proportion import proportion_confint
from sklearn.metrics import cohen_kappa_score

def adjudication_metrics(adj: pd.DataFrame, n_algo_positive_total: int) -> dict:
    pos = adj[adj["algo_positive"]]

    # PPV among algorithm-positives, with a Wilson 95% interval (handles small/extreme counts).
    n_pos = len(pos)
    n_conf = int(pos["confirmed"].sum())
    ppv = n_conf / n_pos
    ppv_lo, ppv_hi = proportion_confint(n_conf, n_pos, alpha=0.05, method="wilson")

    # Inter-rater reliability on the two independent pre-tiebreak calls.
    kappa = cohen_kappa_score(pos["reviewer1_call"].astype(int),
                              pos["reviewer2_call"].astype(int))

    # Survey-weighted PPV when sampling fractions differ by stratum.
    w = pos["samp_weight"]
    ppv_weighted = float((pos["confirmed"] * w).sum() / w.sum())

    # PPV-only bias correction: expected number of TRUE events among all algorithm-positives.
    # (Corrects for false positives; does NOT recover events the algorithm missed -- that needs sensitivity
    #  from a reviewed sample of algorithm-negatives.)
    corrected_true_events = n_algo_positive_total * ppv_weighted

    return {
        "n_reviewed_positive": n_pos,
        "ppv": round(ppv, 4),
        "ppv_wilson_95ci": (round(ppv_lo, 4), round(ppv_hi, 4)),
        "ppv_weighted": round(ppv_weighted, 4),
        "cohen_kappa": round(kappa, 4),
        "corrected_true_event_count": round(corrected_true_events, 1),
    }

library(dplyr)

adjudication_metrics <- function(adj, n_algo_positive_total) {
  pos <- adj %>% filter(algo_positive)

  n_pos  <- nrow(pos)
  n_conf <- sum(pos$confirmed)
  ppv    <- n_conf / n_pos

  # Exact (Clopper-Pearson) PPV interval -- preferred when confirmed counts are small or near 0/1.
  ci <- binom.test(n_conf, n_pos)$conf.int

  # Cohen's kappa on the two independent pre-tiebreak reviewer calls.
  kappa <- psych::cohen.kappa(cbind(as.integer(pos$reviewer1_call),
                                    as.integer(pos$reviewer2_call)))$kappa

  # Survey-weighted PPV (inverse sampling fraction by stratum).
  ppv_weighted <- sum(pos$confirmed * pos$samp_weight) / sum(pos$samp_weight)

  # PPV-only bias correction: expected true events among all algorithm-positives.
  corrected_true_events <- n_algo_positive_total * ppv_weighted

  list(
    n_reviewed_positive   = n_pos,
    ppv                   = round(ppv, 4),
    ppv_exact_95ci        = round(ci, 4),
    ppv_weighted          = round(ppv_weighted, 4),
    cohen_kappa           = round(kappa, 4),
    corrected_true_events = round(corrected_true_events, 1)
  )
}