Predictive and Causal ML Models

In plain language

Machine learning (ML) algorithms can be used for two very different jobs in health research, and mixing them up is the most common mistake in the field. A predictive ML model asks: who is likely to have a bad outcome? It learns patterns in the data to identify high-risk patients, and being accurate is the only goal. A causal ML model asks a harder question: does this treatment actually cause a better outcome? Answering that requires extra steps to control for the fact that sicker patients tend to get certain drugs, and a model that predicts outcomes well does not automatically tell you anything about what caused them.

Modern RWE uses machine learning for two jobs that look similar in code but are governed by completely different statistical logic, and conflating them is the single most common error. Predictive ML (random survival forests, super learner ensembles, gradient-boosted trees, penalized regression) optimizes discrimination and calibration for a target label — 1-year mortality, real-world progression, 30-day readmission — and is judged by AUC, the Integrated Calibration Index (ICI), and calibration slope/intercept. Causal ML (double/debiased machine learning [DML], targeted maximum likelihood estimation [TMLE], honest causal forests) does not optimize prediction of the outcome; it targets a pre-specified causal parameter (ATE, ATT, a survival curve under a sustained strategy, or a conditional average treatment effect [CATE]) and uses ML only to estimate the nuisance functions — the propensity score e(X)=P(A=1|X) and the outcome regressions m_a(X)=E[Y|A=a,X] — which are then combined through an estimator whose bias is first-order insensitive to nuisance error.

Core conceptual distinction

. The estimand, not the algorithm, is the dividing line. A random forest that predicts the outcome with AUC 0.85 tells you nothing about a treatment effect; a poorly discriminating propensity model can still yield a valid ATE if it is well calibrated. Causal ML earns its validity from two structural features. (1) Cross-fitting (sample splitting): nuisances are fit on one partition and evaluated on the held-out partition, so overfitting in the flexible learner does not contaminate the target estimate — naive "fit ML on everyone, then regress" is biased. (2) Neyman orthogonality / double robustness: DML residualizes both treatment and outcome (the Robinson partialling-out / AIPW form) so the score has zero derivative with respect to the nuisances at the truth, giving √n-consistent, asymptotically normal effect estimates when each nuisance converges merely faster than n^(-1/4); TMLE achieves the same by fluctuating an initial outcome fit along the efficient influence function so the plug-in solves the EIF estimating equation and is consistent if either nuisance is correct. Predictive ML has no such guarantee and should never be read as a causal statement (variable importance ≠ causal importance).

Pros, cons, and trade-offs

(named against the alternatives this catalog carries). - vs Cox proportional-hazards regression (`cox-ph-regression`): RSF and causal forests relax proportional hazards and linearity, ingest hundreds of claims codes without manual specification, and deliver heterogeneity (CATE) with honest inference. Cost: no single interpretable hazard ratio, lower regulatory familiarity, and calibration/overlap diagnostics that are harder to communicate. Prefer Cox for a simple pre-specified primary HR with credible PH; prefer the ML family when the confounding surface is high-dimensional or non-linear, PH is implausible, or heterogeneity is the question. - vs propensity-score methods / hdPS (`propensity-score-methods-psm-iptw`, `high-dimensional-propensity-score-hdps-rwe`): DML/TMLE use the same propensity object but pair it with an outcome model and an orthogonal score, so they are doubly robust — hdPS+IPTW relies on the propensity alone and breaks if it is misspecified. Cost: more moving parts, more compute, and the need to defend two nuisance models instead of one. Prefer plain hdPS+IPTW when the team needs a transparent, single-model weighting workflow; prefer DML/TMLE when you want insurance against propensity misspecification. - vs g-estimation / structural nested models (`g-estimation-structural-nested-models`): both are semiparametric, but causal ML targets marginal/CATE parameters with mature software and scales to very high-dimensional baseline confounding, whereas g-estimation directly parameterizes the blip and is the natural tool when effect modification by time-varying factors under treatment-confounder feedback is the scientific target. DML/TMLE for sustained time-varying regimes (LTMLE) exist but are less turnkey. - vs logistic / Poisson regression (`logistic-regression-for-binary-outcomes`, `poisson-negative-binomial-count-models`): ML nuisances capture interactions and non-linearities those parametric models miss, and causal ML returns effects on the risk-difference or risk-ratio scale; cost is the loss of a clean OR/IRR and a heavier validation burden. Parametric models remain superior for moderate dimensions and pre-specified, easily communicated primary models.

When to use

. (1) The confounding surface is genuinely high-dimensional or non-linear (thousands of ICD-10/CPT/NDC proxies, labs, utilization) and parametric specification is guesswork. (2) You want double robustness — insurance against misspecifying the propensity or the outcome model. (3) The estimand is a marginal ATE/ATT or a survival curve under intervention and you want efficient, EIF-based inference (TMLE/DML). (4) Pre-specified heterogeneity: honest causal forests for CATE with valid confidence intervals. (5) As the nuisance engine inside a target-trial emulation (`target-trial-emulation`) or after clone-censor-weight (`clone-censor-weight-per-protocol`).

When NOT to use — and when it is actively misleading or dangerous

. (1) No cross-fitting — fitting flexible ML on the full sample and then regressing reintroduces regularization/overfitting bias; the nominal CIs are anticonservative and the point estimate is attenuated toward or away from the null unpredictably. (2) Positivity/overlap violation — when a drug is reserved for sicker or renally-impaired patients, fitted propensities pile up near 0/1; IPW-style and AIPW estimators then divide by near-zero, variance explodes, and the estimator silently extrapolates into regions with no comparator data. Trim, use overlap weights, or report the restricted estimand — do not present a stabilized point estimate from a cohort with effective sample size collapsed by a handful of extreme weights. (3) Uncalibrated nuisances — efficiency and the TMLE plug-in degrade when ML probabilities are miscalibrated; recalibrate (isotonic/ Platt) before use. (4) Using predictive ML as a causal claim — a high-importance feature in a survival forest is not a causal driver; reporting it as one is the most dangerous failure because it is invisible to standard fit metrics. (5) Temporal leakage — if any "baseline" feature is constructed from post-index claims, the model predicts the future from the future and every effect estimate is corrupted regardless of the orthogonality machinery.

Data-source operational depth

. - Claims (FFS vs Medicare Advantage vs commercial): the engine of high-dimensional confounding control is the lookback feature matrix — every diagnosis/procedure/pharmacy code in a fixed 6–12 month window before index, as ever/never flags, counts, and recency. Two failure modes dominate. First, MA-only person-time lacks complete encounter capture: a patient enrolled only in Medicare Advantage may have no fee-for-service claims, so "absence of a comorbidity code" is missingness, not absence — the ML learns a spurious low-risk signature for MA enrollees. Restrict to A/B/D (or commercial medical+pharmacy) coverage spanning the full lookback and exclude MA-only spans, or add a coverage indicator and never interpret it causally. Second, healthy-user proxies (preventive-service codes, low prior utilization, flu vaccination) are extremely predictive of treatment and partially proxy unmeasured health-seeking behavior (`healthy-user-bias`); they can sharpen confounding control or, if near-deterministic of treatment, push the propensity to the positivity boundary. Pair the final causal estimate with negative-control outcomes (`negative-control-outcomes-rwe`) and an E-value (`e-value-sensitivity-analysis`). - EHR: richer nuisances (labs, vitals, NLP-derived problems) but irregular, visit-driven measurement creates informative missingness and informative loss to follow-up; encode missingness explicitly (indicator + last-value or a dedicated missingness model) rather than letting the learner impute by omission. Calibration drifts across sites and over calendar time, so validate calibration within site/era, not just pooled. - Registry: lower dimension but adjudicated, high-quality labels (stage, recurrence date, graded AE) — ideal for training or validating claims-based predictive models, less so as the sole confounding substrate. Link to claims for complete pharmacy/procedure history and to a death index for censoring. - Linked claims–EHR–vital records: the ideal substrate (EHR severity + claims completeness + reliable mortality), but linkage selects the linkable subset and introduces order/fill/service date discrepancies that must be reconciled before time-zero assignment — otherwise immortal time (`immortal-time-bias-handling`) is silently baked into the feature window.

Worked claims example (DML for the ATT of a new oral oncology agent on 1-year mortality)

Question: among adults with metastatic disease initiating a new oral agent (arm A) versus a clinically interchangeable comparator (arm B), what is the ATT on 1-year all-cause mortality? (1) Cohort: new users of A or B with ≥365 days of continuous A/B/D FFS enrollment before the first qualifying NDC fill (`fill_date`); exclude MA-only spans so the lookback is observable; index_date = first fill; arm from the NDC dispensed that day. (2) Feature matrix X: in [index_date−365, index_date], build ever/never + counts for every ICD-10, CPT, and drug-class code, plus prior lines of therapy, metastasis codes, biomarker-testing codes (PD-L1/MSI), and visit intensity — strictly no post-index codes (leakage guard). (3) Outcome Y: death within 365 days from a mortality-source hierarchy (`mortality-source-hierarchy-rwe`), with administrative censoring at disenrollment/data-end handled by an IPCW term or a survival-DML variant. (4) Cross-fit: 5 folds; on the training folds fit ê(X) (gradient-boosted propensity) and m̂_0, m̂_1 (gradient-boosted outcome); predict on the held-out fold only. (5) Orthogonal score (AIPW/EIF for the ATT): on each held-out fold compute the augmented influence-function contribution and average, weighting toward the treated distribution. (6) Diagnostics: plot the ê(X) overlap by arm, report effective sample size and the fraction with ê>0.95 or <0.05, recalibrate nuisances if ICI is poor, trim or overlap-weight at the boundary, and report an E-value plus a negative-control-outcome estimate. The orthogonality guarantees √n inference even though the gradient-boosting nuisances converge slower than √n — but only because steps (4)–(6) were executed; skip cross-fitting or overlap checks and the entire guarantee evaporates.

Interpreting the output

In the worked example, Drug A is preferentially prescribed to sicker patients; a predictive model correctly learns this pattern and assigns higher outcome risk to Drug A users — but this prediction reflects confounding by severity, not a causal drug effect.

(1) Formal interpretation. A predictive ML model outputs a risk score — the probability that a patient experiences the outcome given their observed features. Variable importance scores rank which features most improve predictive accuracy (e.g., Shapley values, permutation importance). These quantities have no causal interpretation: if severity-of-illness codes are the strongest predictors, that means severity predicts outcomes well in this dataset, not that severity is the dominant modifiable cause. Using a predictive model's risk score as a treatment-effect estimate — for example, comparing mean predicted scores across treatment arms — yields a biased estimate of the causal effect because the score absorbs the confounding structure, not just the treatment signal. Doubly robust causal ML (e.g., DML/AIPW) uses cross-fitting to separate nuisance estimation (propensity, outcome) from effect estimation, providing √n-consistent estimates of the ATT under stated assumptions.

(2) Practical interpretation. When the goal is resource targeting (who will have the outcome?), predictive ML is appropriate. When the goal is intervention evaluation (will treating these patients help?), a causal estimator is required — prediction alone cannot answer it. Treat variable importance rankings as hypothesis-generating signals about predictive utility, not as evidence of causal pathways. If a predictive model is repurposed for policy, document the distinction explicitly and apply a rigorous causal framework with overlap diagnostics, effect estimates, and sensitivity analyses.

Worked example

Scenario

A research team has claims data on 1,000 patients with metastatic cancer who started either Drug A or Drug B. They want to know two things: (1) Can we predict which patients will die within one year? (2) Does Drug A cause lower one-year mortality than Drug B? These look similar but require completely different analytic strategies.

Dataset

Simplified summary of what both questions start from: one row per patient with treatment arm, a severity score, and the one-year outcome.

Steps

• Notice that Drug A patients tend to have higher severity scores (7-9) while Drug B patients tend to have lower scores (3-4). Sicker patients were steered toward Drug A by their doctors.

• A predictive ML model trained on this data will learn that drug=A and high severity both predict death. It will score new patients accurately. That is its only job, and it does it well.

• But if you read the model's output as causal evidence, you would wrongly conclude Drug A kills patients. In reality, the sicker patients who received Drug A would likely have had worse outcomes regardless of which drug they received.

• A causal ML model (such as double ML) explicitly accounts for the fact that treatment assignment was not random. It fits one model to estimate who was likely to receive Drug A given their severity (the propensity), and a second model to estimate what the outcome would have been under each drug separately. It then combines these through a special estimator that is insensitive to errors in either model.

• The result from the causal ML model is an estimate of the treatment effect after removing the influence of severity and other confounders. The predictive ML model cannot produce this estimate, no matter how accurate its predictions are.

Result

Use a predictive ML model when the goal is to score or stratify patients by outcome risk. Use a causal ML model when the goal is to estimate whether a specific treatment causes a difference in outcomes. A high prediction accuracy (AUC) from a predictive model is not evidence of a causal effect and should never be interpreted as one.

Runnable example

import numpy as np
import pandas as pd
from doubleml import DoubleMLData, DoubleMLIRM
from sklearn.ensemble import HistGradientBoostingClassifier

COVARS = [c for c in df.columns if c not in ("person_id", "treat", "outcome")]

# Interactive regression model (IRM) targets the ATE with a doubly robust, cross-fit AIPW score.
dml_data = DoubleMLData(df, y_col="outcome", d_cols="treat", x_cols=COVARS)

ml_g = HistGradientBoostingClassifier(max_depth=4, learning_rate=0.05)  # outcome nuisance m_a(X)
ml_m = HistGradientBoostingClassifier(max_depth=4, learning_rate=0.05)  # propensity nuisance e(X)

dml_irm = DoubleMLIRM(dml_data, ml_g=ml_g, ml_m=ml_m,
                      n_folds=5, n_rep=2, score="ATE", trimming_threshold=0.02)
dml_irm.fit()
print(dml_irm.summary)  # coef = ATE on the risk-difference scale, with cross-fit SE / CI

# Overlap / positivity diagnostics on the cross-fit propensity (do this BEFORE trusting the estimate).
e_hat = dml_irm.predictions["ml_m"].mean(axis=2).ravel()  # averaged over repetitions
ess = (e_hat.sum() ** 2) / np.square(e_hat).sum()         # effective sample size proxy
extreme = np.mean((e_hat < 0.05) | (e_hat > 0.95))
print(f"ESS proxy={ess:.0f}  fraction extreme propensity={extreme:.3f}")

library(DoubleML); library(mlr3); library(mlr3learners)

covars <- setdiff(names(dat), c("treat", "outcome"))
dml_data <- DoubleMLData$new(dat, y_col = "outcome", d_cols = "treat", x_cols = covars)

ml_g <- lrn("regr.ranger",   num.trees = 500)  # outcome nuisance
ml_m <- lrn("classif.ranger", num.trees = 500, predict_type = "prob")  # propensity nuisance

dml_irm <- DoubleMLIRM$new(dml_data, ml_g = ml_g, ml_m = ml_m,
                           n_folds = 5, n_rep = 2, score = "ATE",
                           trimming_threshold = 0.02)
dml_irm$fit()
print(dml_irm$summary())  # ATE on risk-difference scale with cross-fit inference

# Secondary: honest causal forest for pre-specified heterogeneity (CATE).
library(grf)
X <- as.matrix(dat[, covars]); W <- dat$treat; Y <- dat$outcome
cf  <- causal_forest(X, Y, W, num.trees = 2000)
ate <- average_treatment_effect(cf, target.sample = "all")  # doubly robust AIPW ATE
print(ate)
test_calibration(cf)  # mean-prediction + differential-prediction calibration test