Chart Abstraction Interrater Reliability

In plain language

Chart abstraction means people read medical records and extract study data from them. Interrater reliability checks whether two people reading the same chart would reach the same answer. If they do not, the study variable is ambiguous before any statistics are run, so the abstraction manual or reviewer training has to be fixed.

Chart abstraction interrater reliability

is the quality-control process for human extraction of data from medical records. It asks whether two or more abstractors, blinded to each other's answers and usually blinded to exposure when outcomes are involved, would extract the same value from the same source packet using the same abstraction manual. In RWE, chart abstraction often supplies the reference standard for validating claims/EHR algorithms, adjudicating endpoints, identifying reasons for treatment discontinuation, or capturing variables that structured data miss. The abstraction is only as credible as the reliability of the people and instrument producing it.

Core conceptual distinction

Interrater reliability is not the same as algorithm accuracy, clinical truth, or final adjudication. Reliability measures consistency among reviewers before reconciliation. A high Cohen's kappa says the reviewers applied the abstraction instructions similarly; it does not prove the instructions measure the correct clinical construct. A low kappa means the abstracted variable is unstable even before it reaches analysis, and no statistical model downstream can repair ambiguity in the source variable. The usual deliverables are percent agreement, Cohen's kappa for binary or nominal categories, weighted kappa for ordinal categories, intraclass correlation for continuous variables, a discrepancy log, and a retraining or adjudication rule.

Pros, cons, and trade-offs

- Duplicate abstraction vs single abstraction: Duplicate abstraction with independent reviewers quantifies reliability and detects ambiguous instructions. It roughly doubles abstraction effort. Single abstraction is cheaper but hides reviewer drift, site-specific interpretation, and borderline cases. Use duplicate abstraction for primary endpoints, validation truth, and variables requiring judgment. - Percent agreement vs kappa: Percent agreement is easy to explain and should always be reported, but it ignores agreement expected by chance. Kappa corrects for chance and is familiar to reviewers. Kappa can look low under extreme prevalence even when agreement is high, so report both metrics and the marginal distribution. - Calibration sample vs continuous monitoring: A pre-launch calibration sample tests whether the manual is usable before full abstraction. It does not detect drift months later. Continuous duplicate review on a small rotating sample is more expensive but catches drift, new edge cases, and site-specific documentation shifts. - Adjudication after disagreement vs majority vote: A tiebreaker adjudicator produces a final variable for analysis, but the pre-tiebreak reliability metric should still be reported. Majority vote is reproducible but can hide systematic misunderstanding if all reviewers share a bad instruction.

When to use

. Use this concept for any chart-derived variable that matters to the study conclusion: validation of claims or EHR phenotypes, outcome adjudication, treatment-line abstraction, reasons for discontinuation, disease severity, progression, response, contraindications, adverse-event causality, and NLP training labels. It is also needed when an LLM-assisted or NLP abstraction system is benchmarked against human labels; the human reference set must have measured reliability.

When NOT to use - and when it is actively misleading

- Do not use kappa as a decorative statistic after abstraction is complete. Reliability should be a gate. If kappa is poor, the abstraction manual, reviewer training, source packet, or variable definition must be revised before scaling. - Do not let final adjudicated agreement replace pre-tiebreak reliability. Once disagreements are forced to a final answer, apparent agreement is artificially perfect. Report the independent-review agreement before adjudication. - Do not apply unweighted kappa to ordered clinical severity without justification. ECOG 0 vs 1 is a smaller disagreement than ECOG 0 vs 4; weighted kappa or ICC-style methods are usually more appropriate. - Do not interpret low kappa blindly under extreme prevalence. If almost every chart is negative, kappa may be low despite high percent agreement. Report prevalence, percent agreement, positive/negative agreement, and why the metric behaves that way. - Do not claim the chart is a perfect gold standard. Medical records can be incomplete, contradictory, and affected by documentation workflow. Reliability quantifies reviewer consistency, not source-document completeness.

Data-source operational depth

- EHR charts: Source packets may include notes, orders, labs, imaging, medication administration records, problem lists, and outside documents scanned as PDFs. Reliability depends on whether abstractors see the same packet, whether notes are redacted consistently, and whether structured fields are shown alongside narrative text. - Claims-linked chart review: Claims identify candidates, but the chart packet provides clinical truth. Retrieval failure can be non-random, especially for deceased patients or outside facilities. Reliability metrics apply only to reviewed charts, so characterize unretrievable records separately. - Registry abstraction: Registry variables may already be abstracted. If a study re-abstracts or audits registry fields, distinguish registry-abstractor reliability from study-reviewer reliability and preserve the abstraction form version. - Linked EHR-claims-registry: The packet can include conflicting evidence. Reliability requires a source precedence manual: which document wins when discharge summary, lab feed, claim, and registry label disagree?

Worked example

A claims algorithm flags possible hospitalized heart-failure events. The study team samples 200 candidate admissions and creates blinded packets containing discharge summary, BNP, echocardiogram, medication administration records, and selected progress notes. Two nurse abstractors independently classify each case as confirmed heart failure, not confirmed, or insufficient information. On the first 40 calibration packets, observed agreement is 0.78 and unweighted kappa is 0.52. The discrepancy log shows reviewers interpreted "insufficient information" differently, so the manual is revised and retraining occurs. On the next 60 packets, observed agreement rises to 0.90 and kappa to 0.76. Only then does full abstraction proceed. Final disagreements go to a cardiologist, but the reported reliability remains the pre-tiebreak kappa from the independent reviews.

Worked example

Scenario

Two abstractors independently review 10 candidate heart-failure admissions. Each records Y if the admission meets the case definition and N otherwise. The team computes percent agreement and Cohen's kappa before a cardiologist resolves disagreements.

Dataset

Independent chart abstraction calls before adjudication

Steps

• Count Observed Agreement From The Diagonal Cells

  5 charts both called Y and 3 charts both called N, so observed agreement is 8/10 = 0.80.

• Compute expected chance agreement from marginal rates. Abstractor 1 says Y on 6/10 and N on 4/10; Abstractor 2 says Y on 6/10 and N on 4/10. Expected agreement = 0.6 x 0.6 + 0.4 x 0.4 = 0.52.

• Cohen's kappa = (0.80 - 0.52) / (1 - 0.52) = 0.28 / 0.48 = 0.58.

• Kappa of 0.58 is below a common 0.60 threshold for substantial agreement, so the team reviews discrepancies C03 and C06 and revises the manual before scaling.

• The final_after_tiebreak labels are used for analysis, but the reliability report uses the independent pre-tiebreak calls.

Result

Observed agreement is 80% and Cohen's kappa is 0.58. The abstraction process needs clarification and retraining before full-scale chart review.

Agreement Table

Caption

2x2 pre-tiebreak reviewer agreement table

Header Row

• Abstractor 2 Y

• Abstractor 2 N

• Row total

Rows

• • Abstractor 1 Y

  • 5

  • 1

  • 6

• • Abstractor 1 N

  • 1

  • 3

  • 4

• • Column total

  • 6

  • 4

  • 10

Runnable example

import pandas as pd
from sklearn.metrics import cohen_kappa_score

def abstraction_reliability(reviews: pd.DataFrame) -> dict:
    wide = reviews.pivot(index="chart_id", columns="reviewer", values="call").dropna()
    if wide.shape[1] != 2:
        raise ValueError("Expected exactly two reviewers for Cohen's kappa")
    r1, r2 = wide.columns
    observed_agreement = (wide[r1] == wide[r2]).mean()
    kappa = cohen_kappa_score(wide[r1], wide[r2])
    tab = pd.crosstab(wide[r1], wide[r2], rownames=[str(r1)], colnames=[str(r2)])
    return {
        "n_duplicate_reviewed": int(len(wide)),
        "observed_agreement": round(float(observed_agreement), 4),
        "cohen_kappa": round(float(kappa), 4),
        "agreement_table": tab,
    }

library(data.table)
library(irr)

abstraction_reliability <- function(reviews) {
  setDT(reviews)
  wide <- dcast(reviews, chart_id ~ reviewer, value.var = "call")
  wide <- na.omit(wide)
  rating_cols <- setdiff(names(wide), "chart_id")
  if (length(rating_cols) != 2L) stop("Expected exactly two reviewers")

  observed_agreement <- mean(wide[[rating_cols[1]]] == wide[[rating_cols[2]]])
  k <- kappa2(wide[, ..rating_cols], weight = "unweighted")
  list(
    n_duplicate_reviewed = nrow(wide),
    observed_agreement = round(observed_agreement, 4),
    cohen_kappa = round(k$value, 4),
    agreement_table = table(wide[[rating_cols[1]]], wide[[rating_cols[2]]])
  )
}