Agreement Statistics: Kappa, ICC, and Bland-Altman

In plain language

Agreement statistics answer a different question from correlation: two measurement methods can move together perfectly (correlation = 1.0) while one consistently reads twice as high as the other, so they cannot be used interchangeably. Cohen's kappa measures how much two reviewers agree on categorical calls — such as "confirmed heart attack" versus "not confirmed" — beyond what pure chance would predict, and is the standard metric reported after endpoint adjudication and algorithm validation. The intraclass correlation coefficient (ICC) extends this idea to continuous measurements from multiple raters, while the Bland-Altman limits-of-agreement plot shows whether a new measurement method differs from a gold standard by clinically acceptable amounts.

Agreement measures a different question from correlation

The single most consequential distinction in measurement validation is that correlation and agreement are fundamentally different quantities that answer different questions. Two continuous instruments can have a Pearson r of 0.99 and yet fail completely as substitutes for each other: if Instrument A consistently returns values exactly twice those of Instrument B, r = 1.0 while every paired measurement disagrees by 100%. This is not a theoretical curiosity — it is a recurring error in the HEOR and pharmacoepidemiology literature when researchers validate claims-derived biomarker proxies against EHR laboratory values, compare chart-abstracted event rates to adjudicated standards, or calibrate NLP pipeline outputs against manual annotation. Correlation describes the strength and direction of co-movement; agreement asks whether two raters or instruments produce interchangeable values. Reporting a high Pearson or Spearman correlation as evidence of agreement between two measurement methods is methodologically indefensible. The correct tool for agreement depends on the scale of the variable being compared.

For categorical variables (confirmed/not confirmed, disease present/absent, three-level severity grades), the foundational agreement measure is Cohen's kappa. For continuous ratings from multiple raters or occasions on the same subjects (pain scores, troponin readings, star ratings), the intraclass correlation coefficient (ICC) is the appropriate agreement measure. For method comparison — asking whether a new assay, a claims-derived proxy, or an algorithm output can substitute for a gold-standard measurement — Bland and Altman's limits-of-agreement plot is the deliverable.

Cohen's kappa for categorical agreement

Percent observed agreement (the fraction of cases where two raters agree) is an inadequate sole metric because raters can agree by chance alone. Two raters each assigning a binary outcome at 90% true prevalence would agree 82% of the time even with no actual information whatsoever. Cohen's kappa corrects for this chance agreement by comparing what was observed to what would be expected if raters made independent decisions based only on their own marginal tendencies:

kappa = (P_o - P_e) / (1 - P_e)

where P_o is the observed proportion of agreement (the diagonal sum divided by total cases) and P_e is the expected proportion of agreement by chance alone, computed from the marginal probabilities of each rater's calls:

P_e = P(R1=Y) P(R2=Y) + P(R1=N) P(R2=N)

Kappa ranges from -1 (perfect systematic disagreement) through 0 (no agreement beyond chance) to 1 (perfect agreement). The Landis and Koch (1977) benchmark scale places kappa < 0.20 as slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, and 0.81–1.00 almost perfect agreement. Regulatory submissions (FDA, EMA) for adjudicated primary endpoints in observational research typically expect kappa ≥ 0.60 before tiebreak as a prerequisite for the adjudication to be considered reliable.

Weighted kappa for ordinal categories. When categories are ordered — three-grade severity classifications, staging systems, or Likert scales — disagreements that are close on the ordinal scale should be penalized less than distant disagreements. Weighted kappa accomplishes this by applying a weight matrix that reduces the contribution of near-misses. With linear weights, a one-grade discordance is penalized proportionally less than a two-grade discordance; with quadratic weights, large discordances are penalized more steeply. The choice of weighting scheme materially affects the resulting kappa and must be pre-specified in the analysis plan.

The kappa paradox under extreme prevalence. Kappa can be misleadingly low when one category strongly dominates the marginals — a phenomenon called the prevalence paradox. If 95% of adjudicated cases are "not a true event" and both raters agree on "not a true event" almost always, observed agreement may be 0.95 while kappa is near 0.10, because P_e is already approximately 0.90. This situation arises directly in endpoint adjudication of rare events (severe bleeding, fatal MI) and in NLP validation on highly imbalanced corpora. In these settings, report both percent agreement and kappa, and explicitly note the base-rate context so that a low kappa is not misread as evidence of poor reliability.

Intraclass correlation coefficient for continuous agreement

The ICC is a family of statistics quantifying agreement among continuous measurements, extending the kappa idea to scales where the magnitude of disagreement matters and not just its presence. The key design choices are the statistical model and the type of agreement being assessed:

• One-way random effects: each subject is rated by a different random set of raters,
• Two-way random effects, absolute agreement: the same set of raters evaluates all
• Two-way random effects, consistency: the same raters evaluate all subjects, but

Choosing the wrong ICC form is a frequent error. Shrout and Fleiss (1979) catalogued six ICC models; for regulatory submissions evaluating inter-abstractor reliability on continuous endpoints, the two-way absolute-agreement ICC with a single measurement (ICC(2,1)) is the most commonly required form. Always report the ICC model chosen, the number of raters, and the 95% confidence interval alongside the point estimate.

Bland-Altman for method comparison

The Bland-Altman plot directly addresses the practical clinical question: "Can I substitute Method B for Method A in clinical practice?" For n paired observations, compute:

• The mean difference (bias): the average of (Method A - Method B). A non-zero bias
• The limits of agreement: bias ± 1.96 * SD(differences), where SD(differences) is the

The plot shows the average of the two methods on the x-axis against the difference (Method A - Method B) on the y-axis, with horizontal reference lines at the bias and both limits of agreement. The interpretation: approximately 95% of future paired differences between the two methods in similar patients are expected to fall within the limits of agreement, assuming differences are normally distributed and do not depend on the magnitude of the measurement. The clinical decision is whether those limits are narrow enough to be acceptable — a judgment that requires a pre-specified maximum acceptable difference, not a statistical test alone.

If the spread of differences increases with the magnitude of the measurement (a funnel shape on the plot), proportional bias is present. In this case, log-transform both measurements before computing the Bland-Altman, and back-transform the limits of agreement into ratio form (for example, "Method A returns values within a factor of 1.15 of Method B for 95% of paired measurements"). A formal test for proportional bias uses the Pearson correlation between the difference (Method A - Method B) and the mean (Method A + Method B)/2; a significant correlation indicates that the limits of agreement vary with the measurement level.

RWE applications

Agreement statistics arise in four recurring contexts in HEOR and pharmacoepidemiology:

1. Inter-abstractor reliability in chart review and endpoint adjudication: a calibration exercise on a shared set of 20–50 records before the full study adjudication begins establishes pre-tiebreak kappa. Regulators use this to assess whether the adjudication committee was consistent; a kappa below 0.40 on the calibration set typically triggers revision of the case definition and adjudication charter before proceeding. 2. Claims-versus-registry variable concordance: when a claims-derived comorbidity flag or disease status variable is compared to a registry adjudicated label for the same patients, kappa (binary) or ICC (ordered severity) quantifies alignment beyond what sensitivity and specificity alone capture. Agreement complements PPV and sensitivity by correcting for prevalence effects. 3. Algorithm-versus-gold-standard beyond PPV: PPV and sensitivity measure case-finding accuracy of a binary classifier; kappa additionally accounts for the chance agreement component, making it a better single metric when comparing multiple competing algorithm specifications during development and when reporting to external stakeholders. 4. Duplicate-coder quality control in NLP validation: training NLP models on annotated data requires inter-annotator agreement on the labeled training corpus before the model is trained. A kappa below 0.60 on a sample of the training labels indicates the annotation scheme is ambiguous and the resulting model will have a ceiling on its accuracy.

Pros, cons, and trade-offs

Cohen's kappa: chance-corrected, interpretable scale from -1 to 1, directly comparable across studies and widely understood by regulatory reviewers. Weighted kappa accommodates ordinal categories. Cons: the prevalence paradox makes kappa misleadingly low at extreme base rates; kappa does not capture the magnitude of disagreement on a continuous scale; applying kappa to ordinal data without weighting discards the ordering information.

ICC: captures both systematic and random rater differences for continuous data; the two-way absolute-agreement form directly answers the question of interchangeability. The 95% confidence interval is directly interpretable and can be pre-specified as a success criterion. Cons: requires the same raters across all subjects for the two-way form; ICC values are inflated when between-subject variability is high even if rater precision is poor, so the ICC for the same rater pair in a heterogeneous vs homogeneous population can differ dramatically.

Bland-Altman: visually communicates both bias and variability in one plot and directly asks the clinical substitutability question. Cons: requires a pre-defined maximum acceptable difference (a clinical judgment, not a statistical one); assumes differences are normally distributed; does not provide a single summary statistic for pre-registering an agreement threshold.

When to use

Use Cohen's kappa for any binary or categorical agreement task: endpoint adjudication, NLP annotation, diagnostic classification quality control, or comparing algorithm output to chart abstraction on a binary endpoint. Pre-specify kappa ≥ 0.60 as the minimum acceptable threshold for regulatory submissions; use weighted kappa for ordinal scales. Use ICC for panels of raters evaluating a continuous outcome (physiological parameters, biomarker values, functional scores) — always specify the ICC form (one-way/two-way, absolute/consistency, single/average measure) and report the 95% confidence interval. Use Bland-Altman when asking "can one continuous measurement method substitute for another?" — not as a correlation substitute.

When NOT to use

Do not use Pearson or Spearman correlation to assess agreement between two measurement methods or two raters measuring the same quantity; a correlation of 0.99 is fully compatible with systematic bias that makes methods non-interchangeable. Do not use kappa as the sole metric when true prevalence is extreme (above 90% or below 10%); complement it with percent agreement and a description of the prevalence context. Do not use ICC (consistency) when systematic rater offsets carry clinical meaning — this choice hides the most dangerous form of rater disagreement. Do not apply Bland-Altman to categorical or ordinal data; use kappa or weighted kappa. Do not interpret limits of agreement as the measurement error of a single instrument; they reflect the total variability of the paired difference, which aggregates both instruments' contributions.

Interpreting the output

From the worked example below: two cardiologists independently adjudicate 100 candidate MACE events from a comparative drug-safety cohort, blinded to treatment assignment. Observed agreement = 0.90. Rater 1's marginal confirm rate = 80/100 = 0.80; Rater 2's marginal confirm rate = 90/100 = 0.90. Expected chance agreement Pe = 0.80 0.90 + 0.20 0.10 = 0.720 + 0.020 = 0.740. Kappa = (0.90 - 0.74)/(1 - 0.74) = 0.16/0.26 ≈ 0.615.

(1) Formal interpretation. Kappa = 0.615 quantifies the proportion of possible non-chance agreement that was actually achieved: the two raters agreed substantially more than would be predicted from their marginal calling rates alone. A kappa of 0 would mean all agreement is explainable by the raters independently applying their own confirm-rate tendencies; a kappa of 1 would mean perfect agreement beyond chance. The Landis-Koch scale places 0.615 in the "substantial agreement" band (0.61–0.80). The observed agreement of 0.90 is meaningfully higher than the chance-expected agreement of 0.74; the 10 discordant cases where Rater 2 confirmed but Rater 1 did not represent the irreducible clinical judgment zone at the boundary of the case definition.

(2) Practical interpretation. A kappa of 0.615 meets the most common regulatory threshold (≥ 0.60) for a pre-tiebreak inter-rater calibration exercise in a clinical events committee. The 10 discordant cases should be reviewed by a third-party tiebreaker, and the specific features driving disagreement — the clinical findings that led Rater 1 to classify as "not confirmed" — should be documented and used to sharpen the case definition charter for the full study adjudication. If kappa were below 0.40, the calibration exercise has failed and the adjudication must be paused to revise the charter before proceeding with the full event set.

Worked example

Scenario

A claims-based comparative drug-safety study uses a clinical events committee to adjudicate candidate MACE (major adverse cardiovascular events) flagged by a claims algorithm. Two cardiologists independently review 100 candidate event packets, each stripped of the patient's drug assignment. The analyst wants to report the pre-tiebreak inter-rater agreement, compute Cohen's kappa to correct for chance, and interpret the result against the regulatory threshold of kappa ≥ 0.60.

Dataset

Two-by-two agreement table for 100 candidate MACE events reviewed independently by two cardiologists blinded to treatment assignment. Cell values are event counts. Rater 2 was more liberal, confirming 90 of 100 events versus 80 for Rater 1.

Steps

• Count agreements on the diagonal: 80 cases where both said confirmed, and 10 cases where both said not confirmed. Total agreements = 80 + 10 = 90 out of 100 events.

• Compute observed proportion agreement: observed_agreement = (80 + 10)/100 = 90/100 = 0.90.

• Compute each rater's marginal confirm rate. Rater 1 confirmed 80 cases: P_R1_confirm = 80/100 = 0.80, P_R1_not = 20/100 = 0.20. Rater 2 confirmed 90 cases: P_R2_confirm = 90/100 = 0.90, P_R2_not = 10/100 = 0.10.

• Compute expected agreement by chance (Pe): if both raters were drawing from their marginal rates independently, the probability both confirm is 0.80 0.90 = 0.720, and the probability both do not confirm is 0.20 0.10 = 0.020. So Pe = 0.720 + 0.020 = 0.740.

• Apply the kappa formula. Numerator (non-chance agreement): kappa_num = 0.90 - 0.74 = 0.16. Denominator (maximum possible non-chance agreement): kappa_den = 1 - 0.74 = 0.26. Kappa = kappa_num/kappa_den = 0.16/0.26 ≈ 0.615.

• Interpret: kappa ≈ 0.615 falls in the 'substantial agreement' band (0.61–0.80) on the Landis-Koch scale and meets the regulatory threshold of ≥ 0.60. The 10 discordant cases (Rater 2 confirmed, Rater 1 did not) represent edge cases to be resolved by the tiebreaker.

Result

observed_agreement = (80 + 10)/100 = 0.90. Pe = 0.80 0.90 + 0.20 0.10 = 0.720 + 0.020 = 0.740. kappa = 0.16/0.26 ≈ 0.615. The inter-rater agreement is substantial, meeting the kappa ≥ 0.60 threshold for endpoint adjudication in a regulatory submission. The 10 discordant events (Rater 2 confirmed, Rater 1 did not) are sent to a third cardiologist tiebreaker.

Runnable example

import numpy as np
import pandas as pd
from sklearn.metrics import cohen_kappa_score

# ── 1. Cohen's kappa: 2x2 adjudication table from worked example ──
# 100 events: a=80 (both confirmed), b=0 (R1 yes/R2 no), c=10 (R1 no/R2 yes), d=10 (both no)
rater1 = [1] * 80 + [0] * 10 + [0] * 10   # confirmed=1, not confirmed=0
rater2 = [1] * 80 + [1] * 10 + [0] * 10

kappa_uw = cohen_kappa_score(rater1, rater2, weights=None)
print(f"Unweighted kappa: {kappa_uw:.4f}")
# Manual verification: Po = 0.90, Pe = 0.80*0.90 + 0.20*0.10 = 0.74
# kappa = (0.90 - 0.74) / (1 - 0.74) = 0.16 / 0.26 = 0.6154

# ── 2. Weighted kappa for ordinal ratings (3-level severity: 0=none, 1=mild, 2=severe) ──
r1_ord = [0, 0, 1, 1, 2, 2, 1, 0, 2, 1]
r2_ord = [0, 1, 1, 2, 2, 2, 0, 0, 2, 1]
kappa_lin  = cohen_kappa_score(r1_ord, r2_ord, weights="linear")
kappa_quad = cohen_kappa_score(r1_ord, r2_ord, weights="quadratic")
print(f"Linear-weighted kappa (ordinal):    {kappa_lin:.4f}")
print(f"Quadratic-weighted kappa (ordinal): {kappa_quad:.4f}")
# Note: quadratic-weighted kappa = ICC for balanced 2-rater designs on ordinal data

# ── 3. ICC: two-way absolute agreement (ICC 2,1) using pingouin ──
import pingouin as pg
# 10 patients rated by 2 raters on a continuous score (e.g., troponin proxy, 0-100)
np.random.seed(42)
n_patients = 10
true_score = np.random.uniform(20, 80, n_patients)
ratings_long = pd.DataFrame({
    "patient": list(range(n_patients)) * 2,
    "rater":   ["R1"] * n_patients + ["R2"] * n_patients,
    "score":   np.concatenate([
        true_score + np.random.normal(0, 3, n_patients),   # Rater 1: small random error
        true_score + np.random.normal(2, 3, n_patients),   # Rater 2: +2 systematic bias
    ]),
})
icc_result = pg.intraclass_corr(
    data=ratings_long,
    targets="patient",   # subject identifier
    raters="rater",      # rater identifier
    ratings="score",     # continuous rating
)
# ICC(2,1): two-way random, single measurement, absolute agreement
icc21 = icc_result[icc_result["Type"] == "ICC2"].iloc[0]
print(f"\nICC(2,1) absolute agreement: {icc21['ICC']:.4f}  "
      f"95% CI [{icc21['CI95%'][0]:.4f}, {icc21['CI95%'][1]:.4f}]")
# ICC(3,1): two-way mixed, single measurement, consistency (rater offset ignored)
icc31 = icc_result[icc_result["Type"] == "ICC3"].iloc[0]
print(f"ICC(3,1) consistency:         {icc31['ICC']:.4f}  "
      f"95% CI [{icc31['CI95%'][0]:.4f}, {icc31['CI95%'][1]:.4f}]")
print("Note: ICC consistency > ICC absolute when systematic rater bias is present (+2 offset here).")

# ── 4. Bland-Altman limits of agreement ──
# Comparing two troponin assays on 12 patients (systematic +3 offset in Assay B)
assay_a = np.array([45, 62, 38, 71, 55, 49, 66, 42, 58, 74, 37, 53], dtype=float)
assay_b = assay_a + np.random.normal(3, 4, len(assay_a))   # Assay B reads ~3 units higher

means = (assay_a + assay_b) / 2
diffs = assay_a - assay_b          # Method A minus Method B convention
bias  = np.mean(diffs)             # negative: Assay A reads lower than Assay B on average
sd_d  = np.std(diffs, ddof=1)      # ddof=1 for sample SD
loa_hi = bias + 1.96 * sd_d
loa_lo = bias - 1.96 * sd_d

print(f"\nBland-Altman (Assay A - Assay B):")
print(f"  Bias (mean difference):    {bias:.2f}")
print(f"  SD of differences:         {sd_d:.2f}")
print(f"  Upper limit of agreement:  {loa_hi:.2f}")
print(f"  Lower limit of agreement:  {loa_lo:.2f}")
print("  Clinical decision: are these limits acceptable for clinical use?")
print("  (A pre-specified maximum acceptable difference is required to answer this.)")

library(irr)

# ── 1. Cohen's kappa (binary adjudication, worked example) ──
rater1 <- c(rep(1, 80), rep(0, 10), rep(0, 10))   # confirmed=1, not confirmed=0
rater2 <- c(rep(1, 80), rep(1, 10), rep(0, 10))

ratings_bin <- cbind(rater1, rater2)
kappa_uw <- kappa2(ratings_bin, weight = "unweighted")
cat(sprintf("Unweighted kappa: %.4f  (z=%.2f, p=%.4f)\n",
            kappa_uw$value, kappa_uw$statistic, kappa_uw$p.value))
# Expected: kappa ~ 0.615 (Po=0.90, Pe=0.74; see worked example)

# ── 2. Weighted kappa for ordinal ratings ──
r1_ord <- c(0, 0, 1, 1, 2, 2, 1, 0, 2, 1)
r2_ord <- c(0, 1, 1, 2, 2, 2, 0, 0, 2, 1)
ratings_ord <- cbind(r1_ord, r2_ord)

kappa_lin  <- kappa2(ratings_ord, weight = "equal")     # linear weights
kappa_quad <- kappa2(ratings_ord, weight = "squared")   # quadratic weights
cat(sprintf("Linear-weighted kappa:    %.4f\n", kappa_lin$value))
cat(sprintf("Quadratic-weighted kappa: %.4f\n", kappa_quad$value))

# ── 3. ICC: two-way absolute agreement and consistency ──
set.seed(42)
n <- 10
true_score <- runif(n, 20, 80)
r1_cont <- true_score + rnorm(n, 0, 3)   # Rater 1: random error only
r2_cont <- true_score + rnorm(n, 2, 3)   # Rater 2: +2 systematic offset + random error
ratings_cont <- cbind(r1_cont, r2_cont)  # irr::icc expects wide format (rows = subjects)

# Two-way random, absolute agreement, single measurement = ICC(2,1)
icc_abs <- icc(ratings_cont, model = "twoway", type = "agreement", unit = "single")
cat(sprintf("\nICC(2,1) absolute agreement: %.4f  95%% CI [%.4f, %.4f]\n",
            icc_abs$value, icc_abs$lbound, icc_abs$ubound))

# Two-way random, consistency, single measurement = ICC(3,1)
icc_con <- icc(ratings_cont, model = "twoway", type = "consistency", unit = "single")
cat(sprintf("ICC(3,1) consistency:         %.4f  95%% CI [%.4f, %.4f]\n",
            icc_con$value, icc_con$lbound, icc_con$ubound))
cat("Note: absolute < consistency when systematic rater bias is present.\n")

# ── 4. Bland-Altman (base R) ──
assay_a <- c(45, 62, 38, 71, 55, 49, 66, 42, 58, 74, 37, 53)
set.seed(7)
assay_b <- assay_a + rnorm(length(assay_a), 3, 4)   # Assay B: ~3 units higher

ba_means <- (assay_a + assay_b) / 2
ba_diffs  <- assay_a - assay_b
bias      <- mean(ba_diffs)
sd_d      <- sd(ba_diffs)         # R's sd() uses n-1 denominator by default
loa_hi    <- bias + 1.96 * sd_d
loa_lo    <- bias - 1.96 * sd_d

cat(sprintf("\nBland-Altman (Assay A - Assay B):\n"))
cat(sprintf("  Bias (mean diff):          %.2f\n", bias))
cat(sprintf("  SD of differences:         %.2f\n", sd_d))
cat(sprintf("  Upper limit of agreement:  %.2f\n", loa_hi))
cat(sprintf("  Lower limit of agreement:  %.2f\n", loa_lo))

# If blandr is installed, use it for a more complete summary:
# library(blandr)
# blandr.statistics(assay_a, assay_b, sig.level = 0.95)
# blandr.draw(assay_a, assay_b)