EHR Data Quality Dimensions for RWE

In plain language

EHR data quality is the practical question of whether a clinical data field is good enough for the specific study decision it is being asked to support. A lab value may be present but in the wrong unit, a diagnosis may be coded but not clinically true, and a missing medication may mean the patient filled it outside the health system. Analysts check dimensions such as completeness, conformance, plausibility, concordance, timeliness, and provenance before trusting the data.

EHR data quality dimensions

are the named failure modes that determine whether electronic health record data are usable for a specific real-world evidence question. The practical frame is not "is this database good?" but "is this variable, during this time window, in this source population, good enough for this estimand?" Kahn et al.'s harmonized framework gives the core vocabulary: conformance (does the data value follow the expected format, vocabulary, and relational rule?), completeness (is the expected data present at the required patient, encounter, site, and time grain?), and plausibility (is the value clinically and temporally believable?). For RWE implementation, analysts usually add closely related working dimensions: concordance across redundant sources, timeliness or data maturity, and provenance or lineage. These additions do not replace the Kahn framework; they turn it into a protocol-ready audit.

Core conceptual distinction

Data quality is not a pre-analysis housekeeping step and it is not a single percentage missingness table. It is an evidence claim about whether absence, presence, timing, and value meaning are interpretable. A laboratory value can be syntactically conformant, clinically plausible, and still useless for an HbA1c endpoint if 45% of results are unmapped local codes at two high-volume sites. A medication order can be complete in the EHR and still fail as an exposure because no linked dispensing or administration confirms initiation. Conversely, a field with modest missingness may be fit for a confounder adjustment if missingness is balanced by arm and captured in a pre-specified missing-data strategy. The dimension is always judged against the analytic use.

Pros, cons, and trade-offs

- Dimension-level audit vs global vendor scorecard: A vendor scorecard is quick and comparable across databases, but it hides whether the exact variable needed by the study is missing, late, unmapped, or site-specific. Dimension-level checks are slower but defensible to FDA, HTA, and manuscript reviewers because every failure is tied to the exposure, outcome, confounder, or follow-up component it threatens. - Conformance-first vs clinical-review-first: Automated conformance checks catch impossible codes, bad dates, unit strings, duplicate keys, and vocabulary drift cheaply. They cannot determine whether a diagnosis was clinically true. Manual review or validation is expensive but necessary for high-stakes outcomes. Prefer automated conformance checks as the first gate, then target manual review to variables where conformance passes but clinical meaning is uncertain. - Completeness threshold vs missingness mechanism: A threshold such as ">=80% populated" is easy to operationalize but can be wrong in both directions. A 95% complete field can be biased if the 5% missingness is concentrated in the exposed arm; a 60% complete field can be usable if missingness is well explained by encounter type and handled in the estimand. Prefer stratified completeness profiles over a single pooled number. - Plausibility checks vs over-cleaning: Removing outliers protects against unit errors and impossible values, but aggressive trimming can discard true severe disease. A creatinine of 18 mg/dL may be real in kidney failure; a creatinine of 884 mg/dL is likely a umol/L unit failure. Use clinical ranges plus unit-aware conversions, not generic numeric winsorization.

When to use

. Use this concept before locking any EHR-based cohort, exposure, outcome, confounder, or endpoint algorithm; when comparing sites in a distributed network; when converting raw EHR feeds to OMOP, PCORnet, or flat analytic tables; when a protocol claims data are fit for purpose; and whenever a source will support a regulatory, HTA, or payer-facing analysis. The audit should happen at the same grain as the analysis: patient-level for cohort membership, encounter-level for utilization, result-level for labs, order/dispense/admin-level for medications, and event-level for outcomes.

When NOT to use - and when it is actively misleading

- Do not treat data quality as question-independent. A source can be excellent for inpatient mortality and weak for outpatient disease control. A generic pass does not authorize every downstream endpoint. - Do not pool checks across sites, calendar eras, or care settings when the analysis depends on them. Site A can carry final LOINC-coded HbA1c results while Site B carries mostly local codes. A pooled completeness rate hides the source of measurement error. - Do not interpret missing EHR data as absence without an observation model. If a patient receives care outside the health system, the EHR is silent. That silence is not "no event," "no drug," or "normal lab." - Do not clean away the clinical signal. Implausibility rules must distinguish impossible values from true extremes. In severe disease and oncology, values that look like outliers may define the outcome. - Do not let conformance checks stand in for validation. A perfectly formatted diagnosis code can still be a rule-out code, historical problem-list carryover, or billing artifact.

Data-source operational depth

- EHR: The dominant threats are encounter-driven capture, site workflow variation, local-code vocabularies, optional structured fields, delayed result finalization, and external-care leakage. Audit by data element and by site: LOINC mapping rate, RxNorm mapping rate, final/amended lab status, duplicate encounter IDs, observation density before index, and time from service to warehouse availability. - Claims: Claims usually have better enrollment-defined observability and adjudicated payment dates, but poorer clinical granularity. Data quality checks focus on benefit completeness, claim reversals, run-out maturity, line-level units, diagnosis position, place of service, and Medicare Advantage or capitated person-time where fee-for-service claims are incomplete. - Registry: Registries often have strong disease-specific variables and adjudicated outcomes, but data quality depends on enrollment rules, abstraction completeness, update lag, and whether variables are abstracted from charts or imported from EHR feeds. Preserve per-variable provenance because imported EHR values and human abstracted values have different error structures. - Linked data: Linkage improves completeness but creates new quality dimensions: linkability selection, duplicate patients across systems, conflicting values across sources, and date reconciliation. A linked lab/outcome/fill record needs a source-precedence rule before it becomes one analysis row.

Worked example

A study wants to use HbA1c control (>=8.0%) as a baseline effect modifier in an EHR-based target trial emulation. The analyst profiles the Measurement table for the 365 days before index. Conformance: 92% of candidate HbA1c rows are mapped to expected LOINC codes, 6% are local codes with "A1C" in the source name, and 2% are unmapped. Units are mostly percent, but one site sends mmol/mol. Completeness: 78% of eligible patients have at least one HbA1c in the lookback, but completeness is 92% at endocrinology sites and 51% at primary-care sites. Plausibility: values range 4.2% to 15.8% after unit conversion; pre-conversion mmol/mol values would have been misclassified as impossible. Concordance: the latest structured lab agrees with the value embedded in a clinic note in 94% of a 100-chart sample. Timeliness: results posted after index in the warehouse are excluded unless specimen collection date is before index. Verdict: use HbA1c as a baseline covariate with site-stratified missingness indicators and a sensitivity analysis restricted to patients with observed HbA1c; do not use it as an eligibility criterion because that would select toward sites and patients with more frequent testing.

Worked example

Scenario

A team wants to use baseline HbA1c from an EHR warehouse as an effect modifier in a diabetes comparative-effectiveness study. They must decide whether the lab field is fit for that use before locking the protocol.

Dataset

Site-level data quality profile for HbA1c in the 365-day baseline window

Steps

• Check conformance first by confirming HbA1c rows use accepted LOINC or audited local codes and carry parseable numeric values.

• Convert the mmol/mol site to percent before applying clinical plausibility limits; otherwise valid mmol/mol values look like impossible percent values.

• Stratify completeness by site instead of relying on the pooled value, because Site B has both lower testing frequency and lower LOINC mapping.

• Check timeliness by using specimen collection date, not warehouse load date, and excluding late-posted records whose collection date is after index.

• Decide whether the field can support the planned role. It is acceptable as an adjustment variable with site-specific missingness handling, but too selection-prone to use as a strict eligibility criterion.

Result

The HbA1c field is usable as a baseline covariate with unit conversion, site indicators, and a sensitivity analysis restricted to patients with observed baseline HbA1c. It is not fit for an eligibility rule because it would preferentially include patients and sites with more frequent laboratory capture.

Runnable example

import numpy as np
import pandas as pd

HBA1C_LOINC = {"4548-4", "17856-6", "4549-2"}

def hba1c_quality_profile(obs: pd.DataFrame, elig: pd.DataFrame) -> pd.DataFrame:
    o = obs.copy()
    o["obs_date"] = pd.to_datetime(o["obs_date"])
    elig = elig.copy()
    elig["index_date"] = pd.to_datetime(elig["index_date"])

    # Candidate rows: standard LOINC or audited local-code/name logic upstream.
    o["is_hba1c"] = o["loinc"].isin(HBA1C_LOINC) | o["local_code"].str.contains("A1C|HBA1C", case=False, na=False)
    lab = o[o["is_hba1c"] & o["status"].isin(["final", "amended"])].merge(elig, on=["person_id", "site"])
    lab = lab[(lab["obs_date"] >= lab["index_date"] - pd.Timedelta(days=365)) &
              (lab["obs_date"] < lab["index_date"])]

    # Unit harmonization: mmol/mol to percent using NGSP approximation.
    lab["value_pct"] = np.where(lab["unit"].str.lower().isin(["mmol/mol", "mmol_mol"]),
                                (lab["value"] + 46.7) / 28.7,
                                lab["value"])
    lab["conformant"] = lab["loinc"].isin(HBA1C_LOINC) & lab["unit"].str.lower().isin(["%", "percent", "mmol/mol", "mmol_mol"])
    lab["plausible"] = lab["value_pct"].between(3.5, 20.0)

    person_hit = lab.groupby(["site", "person_id"]).agg(
        any_hba1c=("value_pct", "size"),
        any_conformant=("conformant", "any"),
        any_plausible=("plausible", "any"),
    ).reset_index()
    den = elig.groupby("site")["person_id"].nunique().rename("eligible")
    num = person_hit.groupby("site").agg(
        with_hba1c=("person_id", "nunique"),
        with_conformant=("any_conformant", "sum"),
        with_plausible=("any_plausible", "sum"),
    )
    out = num.join(den)
    out["completeness"] = out["with_hba1c"] / out["eligible"]
    out["conformance_among_observed"] = out["with_conformant"] / out["with_hba1c"]
    out["plausibility_among_observed"] = out["with_plausible"] / out["with_hba1c"]
    return out.reset_index()

library(data.table)

hba1c_quality_profile <- function(obs, elig) {
  setDT(obs); setDT(elig)
  hba1c_loinc <- c("4548-4", "17856-6", "4549-2")

  obs[, is_hba1c := loinc %chin% hba1c_loinc | grepl("A1C|HBA1C", local_code, ignore.case = TRUE)]
  lab <- merge(obs[is_hba1c == TRUE & status %chin% c("final", "amended")],
               elig, by = c("person_id", "site"))
  lab <- lab[obs_date >= index_date - 365 & obs_date < index_date]

  lab[, value_pct := fifelse(tolower(unit) %chin% c("mmol/mol", "mmol_mol"),
                             (value + 46.7) / 28.7, value)]
  lab[, conformant := loinc %chin% hba1c_loinc &
         tolower(unit) %chin% c("%", "percent", "mmol/mol", "mmol_mol")]
  lab[, plausible := value_pct >= 3.5 & value_pct <= 20.0]

  per_person <- lab[, .(any_conformant = any(conformant),
                        any_plausible = any(plausible)), by = .(site, person_id)]
  den <- elig[, .(eligible = uniqueN(person_id)), by = site]
  num <- per_person[, .(with_hba1c = uniqueN(person_id),
                        with_conformant = sum(any_conformant),
                        with_plausible = sum(any_plausible)), by = site]
  out <- merge(num, den, by = "site")
  out[, completeness := with_hba1c / eligible]
  out[, conformance_among_observed := with_conformant / with_hba1c]
  out[, plausibility_among_observed := with_plausible / with_hba1c]
  out[]
}