EHR Data Quality Dimensions for RWE

EHR data quality dimensions

are the named failure modes that determine whether electronic health record data are usable for a specific real-world evidence question. The practical frame is not "is this database good?" but "is this variable, during this time window, in this source population, good enough for this estimand?" Kahn et al.'s harmonized framework gives the core vocabulary: conformance (does the data value follow the expected format, vocabulary, and relational rule?), completeness (is the expected data present at the required patient, encounter, site, and time grain?), and plausibility (is the value clinically and temporally believable?).

For RWE implementation, analysts usually add closely related working dimensions: concordance across redundant sources, timeliness or data maturity, and provenance or lineage. These additions do not replace the Kahn framework; they turn it into a protocol-ready audit.

Core conceptual distinction

Data quality is not a pre-analysis housekeeping step and it is not a single percentage missingness table. It is an evidence claim about whether absence, presence, timing, and value meaning are interpretable. A laboratory value can be syntactically conformant, clinically plausible, and still useless for an HbA1c endpoint if 45% of results are unmapped local codes at two high-volume sites.

A medication order can be complete in the EHR and still fail as an exposure because no linked dispensing or administration confirms initiation. Conversely, a field with modest missingness may be fit for a confounder adjustment if missingness is balanced by arm and captured in a pre-specified missing-data strategy. The dimension is always judged against the analytic use.

Pros, cons, and trade-offs

• Dimension-level audit vs global vendor scorecard: A vendor scorecard is quick and comparable across databases, but it hides whether the exact variable needed by the study is missing, late, unmapped, or site-specific. Dimension-level checks are slower but defensible to FDA, HTA, and manuscript reviewers because every failure is tied to the exposure, outcome, confounder, or follow-up component it threatens.
• Conformance-first vs clinical-review-first: Automated conformance checks catch impossible codes, bad dates, unit strings, duplicate keys, and vocabulary drift cheaply. They cannot determine whether a diagnosis was clinically true. Manual review or validation is expensive but necessary for high-stakes outcomes. Prefer automated conformance checks as the first gate, then target manual review to variables where conformance passes but clinical meaning is uncertain.
• Completeness threshold vs missingness mechanism: A threshold such as ">=80% populated" is easy to operationalize but can be wrong in both directions. A 95% complete field can be biased if the 5% missingness is concentrated in the exposed arm; a 60% complete field can be usable if missingness is well explained by encounter type and handled in the estimand. Prefer stratified completeness profiles over a single pooled number.
• Plausibility checks vs over-cleaning: Removing outliers protects against unit errors and impossible values, but aggressive trimming can discard true severe disease. A creatinine of 18 mg/dL may be real in kidney failure; a creatinine of 884 mg/dL is likely a umol/L unit failure. Use clinical ranges plus unit-aware conversions, not generic numeric winsorization.

When NOT to use - and when it is actively misleading

• Do not treat data quality as question-independent. A source can be excellent for inpatient mortality and weak for outpatient disease control. A generic pass does not authorize every downstream endpoint.
• Do not pool checks across sites, calendar eras, or care settings when the analysis depends on them. Site A can carry final LOINC-coded HbA1c results while Site B carries mostly local codes. A pooled completeness rate hides the source of measurement error.
• Do not interpret missing EHR data as absence without an observation model. If a patient receives care outside the health system, the EHR is silent. That silence is not "no event," "no drug," or "normal lab."
• Do not clean away the clinical signal. Implausibility rules must distinguish impossible values from true extremes. In severe disease and oncology, values that look like outliers may define the outcome.
• Do not let conformance checks stand in for validation. A perfectly formatted diagnosis code can still be a rule-out code, historical problem-list carryover, or billing artifact.

Data-source operational depth

• EHR: The dominant threats are encounter-driven capture, site workflow variation, local-code vocabularies, optional structured fields, delayed result finalization, and external-care leakage. Audit by data element and by site: LOINC mapping rate, RxNorm mapping rate, final/amended lab status, duplicate encounter IDs, observation density before index, and time from service to warehouse availability.
• Claims: Claims usually have better enrollment-defined observability and adjudicated payment dates, but poorer clinical granularity. Data quality checks focus on benefit completeness, claim reversals, run-out maturity, line-level units, diagnosis position, place of service, and Medicare Advantage or capitated person-time where fee-for-service claims are incomplete.
• Registry: Registries often have strong disease-specific variables and adjudicated outcomes, but data quality depends on enrollment rules, abstraction completeness, update lag, and whether variables are abstracted from charts or imported from EHR feeds. Preserve per-variable provenance because imported EHR values and human abstracted values have different error structures.
• Linked data: Linkage improves completeness but creates new quality dimensions: linkability selection, duplicate patients across systems, conflicting values across sources, and date reconciliation. A linked lab/outcome/fill record needs a source-precedence rule before it becomes one analysis row.

Worked example

A study wants to use HbA1c control (>=8.0%) as a baseline effect modifier in an EHR-based target trial emulation. The analyst profiles the Measurement table for the 365 days before index. Conformance: 92% of candidate HbA1c rows are mapped to expected LOINC codes, 6% are local codes with "A1C" in the source name, and 2% are unmapped. Units are mostly percent, but one site sends mmol/mol.

Completeness: 78% of eligible patients have at least one HbA1c in the lookback, but completeness is 92% at endocrinology sites and 51% at primary-care sites. Plausibility: values range 4.2% to 15.8% after unit conversion; pre-conversion mmol/mol values would have been misclassified as impossible. Concordance: the latest structured lab agrees with the value embedded in a clinic note in 94% of a 100-chart sample.

Timeliness: results posted after index in the warehouse are excluded unless specimen collection date is before index. Verdict: use HbA1c as a baseline covariate with site-stratified missingness indicators and a sensitivity analysis restricted to patients with observed HbA1c; do not use it as an eligibility criterion because that would select toward sites and patients with more frequent testing.

Decision diagram

flowchart TD
  Source[Raw EHR / claims / registry source] --> Conform[Conformance<br/>valid code, unit, date, key]
  Source --> Complete[Completeness<br/>expected rows present by patient/site/time]
  Source --> Plaus[Plausibility<br/>clinical range and temporal logic]
  Source --> Concord[Concordance<br/>agreement with redundant source]
  Source --> Time[Timeliness<br/>data maturity and late arrivals]
  Source --> Prov[Provenance<br/>source lineage and transformation]
  Conform --> Verdict[Fit-for-purpose verdict<br/>for one estimand]
  Complete --> Verdict
  Plaus --> Verdict
  Concord --> Verdict
  Time --> Verdict
  Prov --> Verdict

Worked example

Scenario

A team wants to use baseline HbA1c from an EHR warehouse as an effect modifier in a diabetes comparative-effectiveness study. They must decide whether the lab field is fit for that use before locking the protocol.

Dataset

Site-level data quality profile for HbA1c in the 365-day baseline window

Steps

Result

The HbA1c field is usable as a baseline covariate with unit conversion, site indicators, and a sensitivity analysis restricted to patients with observed baseline HbA1c. It is not fit for an eligibility rule because it would preferentially include patients and sites with more frequent laboratory capture.

Trade-offs

Runnable example

import numpy as np
import pandas as pd

HBA1C_LOINC = {"4548-4", "17856-6", "4549-2"}

def hba1c_quality_profile(obs: pd.DataFrame, elig: pd.DataFrame) -> pd.DataFrame:
    o = obs.copy()
    o["obs_date"] = pd.to_datetime(o["obs_date"])
    elig = elig.copy()
    elig["index_date"] = pd.to_datetime(elig["index_date"])

    # Candidate rows: standard LOINC or audited local-code/name logic upstream.
    o["is_hba1c"] = o["loinc"].isin(HBA1C_LOINC) | o["local_code"].str.contains("A1C|HBA1C", case=False, na=False)
    lab = o[o["is_hba1c"] & o["status"].isin(["final", "amended"])].merge(elig, on=["person_id", "site"])
    lab = lab[(lab["obs_date"] >= lab["index_date"] - pd.Timedelta(days=365)) &
              (lab["obs_date"] < lab["index_date"])]

    # Unit harmonization: mmol/mol to percent using NGSP approximation.
    lab["value_pct"] = np.where(lab["unit"].str.lower().isin(["mmol/mol", "mmol_mol"]),
                                (lab["value"] + 46.7) / 28.7,
                                lab["value"])
    lab["conformant"] = lab["loinc"].isin(HBA1C_LOINC) & lab["unit"].str.lower().isin(["%", "percent", "mmol/mol", "mmol_mol"])
    lab["plausible"] = lab["value_pct"].between(3.5, 20.0)

    person_hit = lab.groupby(["site", "person_id"]).agg(
        any_hba1c=("value_pct", "size"),
        any_conformant=("conformant", "any"),
        any_plausible=("plausible", "any"),
    ).reset_index()
    den = elig.groupby("site")["person_id"].nunique().rename("eligible")
    num = person_hit.groupby("site").agg(
        with_hba1c=("person_id", "nunique"),
        with_conformant=("any_conformant", "sum"),
        with_plausible=("any_plausible", "sum"),
    )
    out = num.join(den)
    out["completeness"] = out["with_hba1c"] / out["eligible"]
    out["conformance_among_observed"] = out["with_conformant"] / out["with_hba1c"]
    out["plausibility_among_observed"] = out["with_plausible"] / out["with_hba1c"]
    return out.reset_index()

library(data.table)

hba1c_quality_profile <- function(obs, elig) {
  setDT(obs); setDT(elig)
  hba1c_loinc <- c("4548-4", "17856-6", "4549-2")

  obs[, is_hba1c := loinc %chin% hba1c_loinc | grepl("A1C|HBA1C", local_code, ignore.case = TRUE)]
  lab <- merge(obs[is_hba1c == TRUE & status %chin% c("final", "amended")],
               elig, by = c("person_id", "site"))
  lab <- lab[obs_date >= index_date - 365 & obs_date < index_date]

  lab[, value_pct := fifelse(tolower(unit) %chin% c("mmol/mol", "mmol_mol"),
                             (value + 46.7) / 28.7, value)]
  lab[, conformant := loinc %chin% hba1c_loinc &
         tolower(unit) %chin% c("%", "percent", "mmol/mol", "mmol_mol")]
  lab[, plausible := value_pct >= 3.5 & value_pct <= 20.0]

  per_person <- lab[, .(any_conformant = any(conformant),
                        any_plausible = any(plausible)), by = .(site, person_id)]
  den <- elig[, .(eligible = uniqueN(person_id)), by = site]
  num <- per_person[, .(with_hba1c = uniqueN(person_id),
                        with_conformant = sum(any_conformant),
                        with_plausible = sum(any_plausible)), by = site]
  out <- merge(num, den, by = "site")
  out[, completeness := with_hba1c / eligible]
  out[, conformance_among_observed := with_conformant / with_hba1c]
  out[, plausibility_among_observed := with_plausible / with_hba1c]
  out[]
}

Citations

1. [1]Kahn MG, Callahan TJ, Barnard J, et al. A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. eGEMs. 2016;4(1):1244.
2. [2]Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. Journal of the American Medical Informatics Association. 2013;20(1):144-151.
3. [3]Food and Drug Administration. Real-World Data: Assessing Electronic Health Records and Medical Claims Data to Support Regulatory Decision-Making for Drug and Biological Products. Guidance for Industry. 2024.

1. [4]Hersh WR, Weiner MG, Embi PJ, et al. Caveats for the use of operational electronic health record data in comparative effectiveness research. Medical Care. 2013;51(8 Suppl 3):S30-S37.