Propensity Score Calibration

In plain language

Propensity score calibration is a way to use a smaller, richer data sample to correct a much larger database study. The big claims study may not know who smokes or has severe disease, but a linked chart-review subset does; PSC learns how those missing clinical factors change the propensity score and uses that information to adjust the main result.

Propensity score calibration (PSC)

is regression calibration applied to a propensity-score problem. The main cohort is large and cheap: claims or EHR data contain treatment, outcome, and a baseline covariate set X, but omit an important confounder C such as smoking, BMI, ECOG performance status, disease severity, lab value, genotype, or frailty. A validation subset is smaller and expensive: it measures X plus C through chart review, linkage, registry abstraction, survey, or enriched EHR extraction. Analysts fit two scores inside the validation subset: an error-prone PS e*(X), using the same variables available for everyone, and a gold-standard PS e(X,C), using the richer covariate set. The calibration model estimates how the gold-standard logit score relates to the error-prone logit score, then uses that relationship to correct the main-study treatment coefficient. PSC is most useful when the missing confounder is unavailable for most patients but is measured well enough in a defensible validation subset to quantify how far the claims-only score is from the clinical score.

Core identification distinction

PSC is not ordinary PS matching or high-dimensional PS. Ordinary PS methods balance measured baseline covariates in the main data only; hdPS adds empirical proxy covariates from pre-index codes. PSC asks a different question: given a validation subset that measures the missing covariate directly, how much did the error-prone PS understate or misdirect confounding adjustment? Its credibility rests on three assumptions. (1) The validation subset is exchangeable with the main cohort for the calibration relationship. (2) The error-prone score is a useful surrogate for the gold-standard score, so residual confounding from C aligns with confounding captured by X rather than pointing in the opposite direction. (3) The validation measurement of C is itself close enough to truth. If these fail, PSC can overcorrect, undercorrect, or move the estimate the wrong way.

Pros, cons, and trade-offs

- vs investigator-specified PS / IPTW / matching: PSC can correct residual confounding from a covariate not measured in the full data, while standard PS methods cannot. Cost: PSC adds a validation-sample transportability assumption and a calibration-model assumption; a clean balance table in the main cohort does not validate the correction. - vs high-dimensional PS: hdPS uses many claims codes as proxies for latent severity; PSC measures a subset of the latent severity variable directly. Cost: hdPS can be run for everyone with no chart review; PSC needs a validation sample and is vulnerable when the validation sample is small or linkable patients differ from non-linkable patients. - vs external adjustment / probabilistic bias analysis: PSC uses individual-level validation data to model the score relationship rather than applying a marginal bias factor from published prevalence and risk-ratio parameters. Cost: the PSC correction is more model-dependent and harder to communicate; probabilistic bias analysis may be more transparent when the validation data are too thin for calibration. - vs multiple imputation of missing confounders: Imputation tries to fill C for every subject, often from auxiliary variables. PSC instead corrects the treatment effect through the score. Imputation can be preferable when C is missing at random and many auxiliary predictors are available; PSC is preferable when the validation subset is explicitly designed to quantify residual confounding and the score-scale correction is the estimand-facing deliverable.

When to use

Use PSC for active-comparator new-user RWE when the main database is fit for treatment/outcome timing but misses a strong baseline confounder that is measured in an internal validation subset or a linkable enriched source. Typical examples are claims studies enriched by EHR labs, BMI, smoking, ECOG, tumor stage, renal function, or disease activity for a sample of patients. Pre-specify the validation sampling frame, the missing covariate set C, the main PS model, the gold-standard PS model, the calibration model scale, and whether the corrected estimand is ATT, ATE, or ATO. Report both the main-data estimate and the PSC-corrected estimate, with uncertainty that includes validation-sample variability.

When NOT to use -- and when it is actively misleading or dangerous

- Validation patients are not transportable. A linked EHR subset may be younger, commercially insured, urban, or treated in academic centers. Applying its calibration relationship to a national Medicare FFS cohort silently treats linkage selection as harmless. - The missing covariate confounds in the opposite direction from X. PSC depends on a surrogacy-style condition. If measured claims variables make treated patients look sicker, but the missing clinical variable makes them healthier, the error-prone PS is a poor surrogate for the gold-standard PS and PSC can increase bias. - The validation subset lacks outcome information when the surrogacy assumption is questionable. Outcome data in the validation subset are what let analysts diagnose whether calibration is likely to help or harm; without it, present a sensitivity range rather than a single corrected estimate. - Exposure or outcome timing is wrong. PSC cannot rescue prevalent-user bias, immortal time, post-index covariates in the PS, or a comparator that is not clinically exchangeable. - The missing covariate is measured after treatment start. Post-index labs, ECOG, or stage updates can be mediators or effects of surveillance. Treating them as baseline confounders creates over-adjustment or collider bias.

Data-source operational depth

- Claims: Claims are usually the main phase because treatment fills, procedures, diagnoses, costs, and enrollment are available for everyone. Require complete observable medical+pharmacy capture and exclude Medicare Advantage-only person-time when baseline covariates are derived from FFS claims. PSC is valuable when C is absent from claims (smoking, BMI, ECOG, lab severity). Do not treat absence of claims codes as absence of disease in incomplete payer segments. - EHR: EHR is often the validation phase because labs, vitals, smoking, BMI, problem-list details, and notes can be abstracted for a subset. Failure modes are visit-driven missingness, site-specific coding, outside-care leakage, and selective chart availability. Include encounter intensity and site in the validation diagnostics. - Registry: Registry data can supply tumor stage, biomarkers, ECOG, procedure details, and adjudicated clinical status. The registry population may not represent all treated patients; linkage eligibility and match failure should be modeled or at least described by arm and key baseline strata. - Linked data: Linked claims-EHR-registry data are ideal for PSC but create a second selection process. Compare linked and unlinked patients on age, payer, geography, utilization, baseline disease, and treatment arm before transporting the calibration equation to the full cohort.

Worked RWE example

A claims study compares Drug A with Drug B for heart-failure hospitalization. The claims-only PS adjusts for age, sex, prior HF, CKD diagnoses, prior inpatient count, and drug history, but not smoking or BMI. A 12% linked EHR validation subset contains smoking, BMI, baseline eGFR, and NYHA class. In the validation subset, analysts fit e*(X) from claims variables and e(X,C) from claims plus EHR variables. The calibration regression shows that high claims-only scores systematically understate treatment probability among patients with obesity and NYHA III/IV heart failure. The claims-only weighted HR is 0.74; after PSC the HR is 0.86 with a wider interval. Interpretation: the apparent benefit was partly residual confounding by severity absent from claims, but the corrected estimate is conditional on the linked EHR subset transporting to the full claims cohort.

Worked example

Scenario

A claims study of Drug A versus Drug B lacks smoking and BMI, but a linked EHR validation subset contains them. The analyst compares the claims-only propensity score with a richer score in the validation subset and uses that calibration relationship to correct the claims-only effect estimate.

Dataset

Validation-subset calibration signal

Steps

• Fit the main PS e*(X) in the full claims cohort and save the logit score for every patient.

• In the validation subset, fit the same e*(X) model and a gold-standard e(X,C) model that adds smoking, BMI, eGFR, and severity.

• Regress the gold-standard logit score on the error-prone logit score and key calibration terms inside the validation subset.

• Apply the calibration equation to full-cohort error-prone scores and re-estimate the treatment effect using the calibrated score or the corrected treatment coefficient.

• Bootstrap the validation subset or use a sandwich variance so the corrected interval reflects calibration uncertainty.

Result

The uncalibrated claims-only HR of 0.74 moves to a PSC-corrected HR of 0.86, indicating that missing severity variables made Drug A look more protective than it was under the validation-subset assumptions.

Runnable example

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression, LinearRegression

def _logit(p):
    p = np.clip(p, 1e-6, 1 - 1e-6)
    return np.log(p / (1 - p))

def fit_psc(main, val, treatment_col, x_cols, c_cols):
    # Error-prone PS: what the full database can measure.
    err = LogisticRegression(max_iter=2000, C=1e6)
    err.fit(main[x_cols], main[treatment_col])
    main = main.copy()
    val = val.copy()
    main["ps_error"] = err.predict_proba(main[x_cols])[:, 1]
    val["ps_error"] = err.predict_proba(val[x_cols])[:, 1]

    # Gold-standard PS: richer validation-only covariates added.
    gold = LogisticRegression(max_iter=2000, C=1e6)
    gold.fit(val[x_cols + c_cols], val[treatment_col])
    val["ps_gold"] = gold.predict_proba(val[x_cols + c_cols])[:, 1]

    # Calibration on the logit scale. Add validation-sampling weights here if the subset was stratified.
    cal = LinearRegression()
    x_val = _logit(val["ps_error"]).to_numpy().reshape(-1, 1)
    y_val = _logit(val["ps_gold"]).to_numpy()
    cal.fit(x_val, y_val)
    x_main = _logit(main["ps_error"]).to_numpy().reshape(-1, 1)
    main["ps_calibrated"] = 1 / (1 + np.exp(-cal.predict(x_main)))
    return main, val, {"intercept": float(cal.intercept_), "slope": float(cal.coef_[0])}

fit_psc <- function(main, val, treatment, x_cols, c_cols) {
  f_err <- as.formula(paste(treatment, "~", paste(x_cols, collapse = " + ")))
  m_err <- glm(f_err, data = main, family = binomial())
  main$ps_error <- predict(m_err, newdata = main, type = "response")
  val$ps_error  <- predict(m_err, newdata = val,  type = "response")

  f_gold <- as.formula(paste(treatment, "~", paste(c(x_cols, c_cols), collapse = " + ")))
  m_gold <- glm(f_gold, data = val, family = binomial())
  val$ps_gold <- predict(m_gold, newdata = val, type = "response")

  logit <- function(p) qlogis(pmin(pmax(p, 1e-6), 1 - 1e-6))
  cal <- lm(logit(ps_gold) ~ logit(ps_error), data = val)
  main$ps_calibrated <- plogis(predict(cal, newdata = main))
  list(main = main, validation = val, calibration = cal)
}