Predictive and Causal ML Models

Modern RWE uses machine learning for two jobs that look similar in code but are governed by completely different statistical logic, and conflating them is the single most common error.

Predictive ML

(random survival forests, super learner ensembles, gradient-boosted trees, penalized regression) optimizes discrimination and calibration for a target label — 1-year mortality, real-world progression, 30-day readmission — and is judged by AUC, the Integrated Calibration Index (ICI), and calibration slope/intercept.

Causal ML

(double/debiased machine learning [DML], targeted maximum likelihood estimation [TMLE], honest causal forests) does not optimize prediction of the outcome; it targets a pre-specified causal parameter (ATE, ATT, a survival curve under a sustained strategy, or a conditional average treatment effect [CATE]) and uses ML only to estimate the nuisance functions — the propensity score e(X)=P(A=1|X) and the outcome regressions m_a(X)=E[Y|A=a,X] — which are then combined through an estimator whose bias is first-order insensitive to nuisance error.

Core conceptual distinction

. The estimand, not the algorithm, is the dividing line. A random forest that predicts the outcome with AUC 0.85 tells you nothing about a treatment effect; a poorly discriminating propensity model can still yield a valid ATE if it is well calibrated.

Causal ML earns its validity from two structural features. (1) Cross-fitting (sample splitting): nuisances are fit on one partition and evaluated on the held-out partition, so overfitting in the flexible learner does not contaminate the target estimate — naive "fit ML on everyone, then regress" is biased.

(2) Neyman orthogonality / double robustness: DML residualizes both treatment and outcome (the Robinson partialling-out / AIPW form) so the score has zero derivative with respect to the nuisances at the truth, giving √n-consistent, asymptotically normal effect estimates when each nuisance converges merely faster than n^(-1/4); TMLE achieves the same by fluctuating an initial outcome fit along the efficient influence function so the plug-in solves the EIF estimating equation and is consistent if either nuisance is correct.

Predictive ML has no such guarantee and should never be read as a causal statement (variable importance ≠ causal importance).

Pros, cons, and trade-offs

(named against the alternatives this catalog carries).

• vs Cox proportional-hazards regression (`cox-ph-regression`): RSF and causal forests relax proportional hazards and linearity, ingest hundreds of claims codes without manual specification, and deliver heterogeneity (CATE) with honest inference. Cost: no single interpretable hazard ratio, lower regulatory familiarity, and calibration/overlap diagnostics that are harder to communicate. Prefer Cox for a simple pre-specified primary HR with credible PH; prefer the ML family when the confounding surface is high-dimensional or non-linear, PH is implausible, or heterogeneity is the question.
• vs propensity-score methods / hdPS (`propensity-score-methods-psm-iptw`, `high-dimensional-propensity-score-hdps-rwe`): DML/TMLE use the same propensity object but pair it with an outcome model and an orthogonal score, so they are doubly robust — hdPS+IPTW relies on the propensity alone and breaks if it is misspecified. Cost: more moving parts, more compute, and the need to defend two nuisance models instead of one. Prefer plain hdPS+IPTW when the team needs a transparent, single-model weighting workflow; prefer DML/TMLE when you want insurance against propensity misspecification.
• vs g-estimation / structural nested models (`g-estimation-structural-nested-models`): both are semiparametric, but causal ML targets marginal/CATE parameters with mature software and scales to very high-dimensional baseline confounding, whereas g-estimation directly parameterizes the blip and is the natural tool when effect modification by time-varying factors under treatment-confounder feedback is the scientific target. DML/TMLE for sustained time-varying regimes (LTMLE) exist but are less turnkey.
• vs logistic / Poisson regression (`logistic-regression-for-binary-outcomes`, `poisson-negative-binomial-count-models`): ML nuisances capture interactions and non-linearities those parametric models miss, and causal ML returns effects on the risk-difference or risk-ratio scale; cost is the loss of a clean OR/IRR and a heavier validation burden. Parametric models remain superior for moderate dimensions and pre-specified, easily communicated primary models.

When NOT to use — and when it is actively misleading or dangerous

.

1. No cross-fitting — fitting flexible ML on the full sample and then regressing reintroduces regularization/overfitting bias; the nominal CIs are anticonservative and the point estimate is attenuated toward or away from the null unpredictably.
2. Positivity/overlap violation — when a drug is reserved for sicker or renally-impaired patients, fitted propensities pile up near 0/1; IPW-style and AIPW estimators then divide by near-zero, variance explodes, and the estimator silently extrapolates into regions with no comparator data. Trim, use overlap weights, or report the restricted estimand — do not present a stabilized point estimate from a cohort with effective sample size collapsed by a handful of extreme weights.
3. Uncalibrated nuisances — efficiency and the TMLE plug-in degrade when ML probabilities are miscalibrated; recalibrate (isotonic/ Platt) before use.
4. Using predictive ML as a causal claim — a high-importance feature in a survival forest is not a causal driver; reporting it as one is the most dangerous failure because it is invisible to standard fit metrics.
5. Temporal leakage — if any "baseline" feature is constructed from post-index claims, the model predicts the future from the future and every effect estimate is corrupted regardless of the orthogonality machinery.

Data-source operational depth

.

• Claims (FFS vs Medicare Advantage vs commercial): the engine of high-dimensional confounding control is the lookback feature matrix — every diagnosis/procedure/pharmacy code in a fixed 6–12 month window before index, as ever/never flags, counts, and recency. Two failure modes dominate. First, MA-only person-time lacks complete encounter capture: a patient enrolled only in Medicare Advantage may have no fee-for-service claims, so "absence of a comorbidity code" is missingness, not absence — the ML learns a spurious low-risk signature for MA enrollees. Restrict to A/B/D (or commercial medical+pharmacy) coverage spanning the full lookback and exclude MA-only spans, or add a coverage indicator and never interpret it causally. Second, healthy-user proxies (preventive-service codes, low prior utilization, flu vaccination) are extremely predictive of treatment and partially proxy unmeasured health-seeking behavior (`healthy-user-bias`); they can sharpen confounding control or, if near-deterministic of treatment, push the propensity to the positivity boundary. Pair the final causal estimate with negative-control outcomes (`negative-control-outcomes-rwe`) and an E-value (`e-value-sensitivity-analysis`).
• EHR: richer nuisances (labs, vitals, NLP-derived problems) but irregular, visit-driven measurement creates informative missingness and informative loss to follow-up; encode missingness explicitly (indicator + last-value or a dedicated missingness model) rather than letting the learner impute by omission. Calibration drifts across sites and over calendar time, so validate calibration within site/era, not just pooled.
• Registry: lower dimension but adjudicated, high-quality labels (stage, recurrence date, graded AE) — ideal for training or validating claims-based predictive models, less so as the sole confounding substrate. Link to claims for complete pharmacy/procedure history and to a death index for censoring.
• Linked claims–EHR–vital records: the ideal substrate (EHR severity + claims completeness + reliable mortality), but linkage selects the linkable subset and introduces order/fill/service date discrepancies that must be reconciled before time-zero assignment — otherwise immortal time (`immortal-time-bias-handling`) is silently baked into the feature window.

Worked claims example (DML for the ATT of a new oral oncology agent on 1-year mortality)

Question: among adults with metastatic disease initiating a new oral agent (arm A) versus a clinically interchangeable comparator (arm B), what is the ATT on 1-year all-cause mortality?

1. Cohort: new users of A or B with ≥365 days of continuous A/B/D FFS enrollment before the first qualifying NDC fill (`fill_date`); exclude MA-only spans so the lookback is observable; index_date = first fill; arm from the NDC dispensed that day.
2. Feature matrix X: in [index_date−365, index_date], build ever/never + counts for every ICD-10, CPT, and drug-class code, plus prior lines of therapy, metastasis codes, biomarker-testing codes (PD-L1/MSI), and visit intensity — strictly no post-index codes (leakage guard).
3. Outcome Y: death within 365 days from a mortality-source hierarchy (`mortality-source-hierarchy-rwe`), with administrative censoring at disenrollment/data-end handled by an IPCW term or a survival-DML variant.
4. Cross-fit: 5 folds; on the training folds fit ê(X) (gradient-boosted propensity) and m̂_0, m̂_1 (gradient-boosted outcome); predict on the held-out fold only.
5. Orthogonal score (AIPW/EIF for the ATT): on each held-out fold compute the augmented influence-function contribution and average, weighting toward the treated distribution.
6. Diagnostics: plot the ê(X) overlap by arm, report effective sample size and the fraction with ê>0.95 or <0.05, recalibrate nuisances if ICI is poor, trim or overlap-weight at the boundary, and report an E-value plus a negative-control-outcome estimate. The orthogonality guarantees √n inference even though the gradient-boosting nuisances converge slower than √n — but only because steps (4)– (6) were executed; skip cross-fitting or overlap checks and the entire guarantee evaporates.

Interpreting the output

In the worked example, Drug A is preferentially prescribed to sicker patients; a predictive model correctly learns this pattern and assigns higher outcome risk to Drug A users — but this prediction reflects confounding by severity, not a causal drug effect.

(1) Formal interpretation. A predictive ML model outputs a risk score — the probability that a patient experiences the outcome given their observed features. Variable importance scores rank which features most improve predictive accuracy (e.g., Shapley values, permutation importance). These quantities have no causal interpretation: if severity-of-illness codes are the strongest predictors, that means severity predicts outcomes well in this dataset, not that severity is the dominant modifiable cause.

Using a predictive model's risk score as a treatment-effect estimate — for example, comparing mean predicted scores across treatment arms — yields a biased estimate of the causal effect because the score absorbs the confounding structure, not just the treatment signal. Doubly robust causal ML (e.g., DML/AIPW) uses cross-fitting to separate nuisance estimation (propensity, outcome) from effect estimation, providing √n-consistent estimates of the ATT under stated assumptions.

(2) Practical interpretation. When the goal is resource targeting (who will have the outcome?), predictive ML is appropriate. When the goal is intervention evaluation (will treating these patients help?), a causal estimator is required — prediction alone cannot answer it. Treat variable importance rankings as hypothesis-generating signals about predictive utility, not as evidence of causal pathways. If a predictive model is repurposed for policy, document the distinction explicitly and apply a rigorous causal framework with overlap diagnostics, effect estimates, and sensitivity analyses.

Decision diagram

flowchart TD
  Q["Question: causal effect or pure prediction?"] -->|prediction only| Pred["Predictive ML<br/>RSF / super learner / GBM"]
  Q -->|causal estimand| Est["Pre-specify estimand<br/>ATE / ATT / CATE / survival curve"]
  Pred --> Cal["Evaluate AUC + ICI / calibration<br/>do NOT read importance as causal"]
  Est --> CF["Cross-fit K folds: fit nuisances on train, predict on holdout"]
  CF --> Nui["Propensity e-hat(X) + outcome m-hat_a(X)<br/>via ML / super learner"]
  Nui --> Ortho["Orthogonal / doubly robust combine<br/>DML AIPW score or TMLE targeting step"]
  Ortho --> Diag["Overlap + ESS + nuisance calibration<br/>trim / overlap weights at boundary"]
  Diag --> Out["Effect + EIF-based CI<br/>+ E-value + negative-control outcome"]

flowchart LR
  Healthy[Unmeasured health-seeking behavior] -->|proxied by| Feat[High-dim claims features<br/>preventive codes, low prior utilization]
  Healthy --> Tx[Treatment choice A vs B]
  Healthy --> Y[Outcome risk]
  Feat --> Tx
  Feat --> Y
  Tx --> Y
  Feat -.near-deterministic of Tx.-> Pos[Positivity boundary<br/>e-hat -> 0 or 1]
  style Healthy fill:#ffcccc
  style Pos fill:#fff2cc

Worked example

Scenario

A research team has claims data on 1,000 patients with metastatic cancer who started either Drug A or Drug B. They want to know two things: (1) Can we predict which patients will die within one year? (2) Does Drug A cause lower one-year mortality than Drug B? These look similar but require completely different analytic strategies.

Dataset

Simplified summary of what both questions start from: one row per patient with treatment arm, a severity score, and the one-year outcome.

Steps

Result

Use a predictive ML model when the goal is to score or stratify patients by outcome risk. Use a causal ML model when the goal is to estimate whether a specific treatment causes a difference in outcomes. A high prediction accuracy (AUC) from a predictive model is not evidence of a causal effect and should never be interpreted as one.

Trade-offs

Runnable example

import numpy as np
import pandas as pd
from doubleml import DoubleMLData, DoubleMLIRM
from sklearn.ensemble import HistGradientBoostingClassifier

COVARS = [c for c in df.columns if c not in ("person_id", "treat", "outcome")]

# Interactive regression model (IRM) targets the ATE with a doubly robust, cross-fit AIPW score.
dml_data = DoubleMLData(df, y_col="outcome", d_cols="treat", x_cols=COVARS)

ml_g = HistGradientBoostingClassifier(max_depth=4, learning_rate=0.05)  # outcome nuisance m_a(X)
ml_m = HistGradientBoostingClassifier(max_depth=4, learning_rate=0.05)  # propensity nuisance e(X)

dml_irm = DoubleMLIRM(dml_data, ml_g=ml_g, ml_m=ml_m,
                      n_folds=5, n_rep=2, score="ATE", trimming_threshold=0.02)
dml_irm.fit()
print(dml_irm.summary)  # coef = ATE on the risk-difference scale, with cross-fit SE / CI

# Overlap / positivity diagnostics on the cross-fit propensity (do this BEFORE trusting the estimate).
e_hat = dml_irm.predictions["ml_m"].mean(axis=2).ravel()  # averaged over repetitions
ess = (e_hat.sum() ** 2) / np.square(e_hat).sum()         # effective sample size proxy
extreme = np.mean((e_hat < 0.05) | (e_hat > 0.95))
print(f"ESS proxy={ess:.0f}  fraction extreme propensity={extreme:.3f}")

library(DoubleML); library(mlr3); library(mlr3learners)

covars <- setdiff(names(dat), c("treat", "outcome"))
dml_data <- DoubleMLData$new(dat, y_col = "outcome", d_cols = "treat", x_cols = covars)

ml_g <- lrn("regr.ranger",   num.trees = 500)  # outcome nuisance
ml_m <- lrn("classif.ranger", num.trees = 500, predict_type = "prob")  # propensity nuisance

dml_irm <- DoubleMLIRM$new(dml_data, ml_g = ml_g, ml_m = ml_m,
                           n_folds = 5, n_rep = 2, score = "ATE",
                           trimming_threshold = 0.02)
dml_irm$fit()
print(dml_irm$summary())  # ATE on risk-difference scale with cross-fit inference

# Secondary: honest causal forest for pre-specified heterogeneity (CATE).
library(grf)
X <- as.matrix(dat[, covars]); W <- dat$treat; Y <- dat$outcome
cf  <- causal_forest(X, Y, W, num.trees = 2000)
ate <- average_treatment_effect(cf, target.sample = "all")  # doubly robust AIPW ATE
print(ate)
test_calibration(cf)  # mean-prediction + differential-prediction calibration test

%let covs = age sex dx_chf dx_ckd rx_insulin_cnt util_visits;  /* pre-index baseline covariates only */

/* ---- One cross-fit fold (k=1 held out). Wrap folds 1..5 in a %macro %do loop in production. ---- */
data train hold;
  set work.analytic;
  if fold = 1 then output hold; else output train;
run;

/* Propensity nuisance e_hat(X) = P(treat=1 | X), trained on the OTHER folds, scored on the held-out fold. */
proc hplogistic data=train;
  class sex dx_chf dx_ckd;
  model treat(event='1') = &covs;
  code file="phat.sas";                 /* exportable scoring code -> applies to held-out rows */
run;
data hold; set hold; %include "phat.sas"; e_hat = P_treat1; run;

/* Outcome nuisances via random forest: m1 = E[Y|A=1,X] and m0 = E[Y|A=0,X], trained on train, scored on hold. */
proc hpforest data=train(where=(treat=1)) maxtrees=300;
  target outcome / level=binary;
  input &covs;
  save file="m1.bin";
run;
proc hp4score data=hold; score file="m1.bin" out=hold1; run;   /* P_outcome1 -> m1_hat */
proc hpforest data=train(where=(treat=0)) maxtrees=300;
  target outcome / level=binary;
  input &covs;
  save file="m0.bin";
run;
proc hp4score data=hold1 file="m0.bin" out=hold0;              /* P_outcome1 -> m0_hat */
run;

/* AIPW / doubly robust influence-function contribution on the held-out fold (orthogonal score). */
data eif;
  set hold0;
  e_hat = min(max(e_hat, 0.02), 0.98);                         /* trim at the positivity boundary */
  psi = (m1_hat - m0_hat)
      + treat*(outcome - m1_hat)/e_hat
      - (1-treat)*(outcome - m0_hat)/(1 - e_hat);
run;

/* Cross-fit ATE = mean of psi pooled over all held-out folds; SE from the EIF (influence-function) variance. */
proc means data=eif n mean stderr clm; var psi; run;

Citations

1. [1]Ishwaran H, Kogalur UB, Blackstone EH, Lauer MS. Random survival forests. The Annals of Applied Statistics. 2008;2(3):841-860.
2. [2]Chernozhukov V, Chetverikov D, Demirer M, Duflo E, Hansen C, Newey W, Robins J. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal. 2018;21(1):C1-C68.
3. [3]van der Laan MJ, Polley EC, Hubbard AE. Super Learner. Statistical Applications in Genetics and Molecular Biology. 2007;6(1):Article 25.

1. [4]Schuler MS, Rose S. Targeted maximum likelihood estimation for causal inference in observational studies. American Journal of Epidemiology. 2017;185(1):65-73.

1. [5]Wager S, Athey S. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association. 2018;113(523):1228-1242.