Cross-Validation, Overfitting, and Optimism

In plain language

A prediction model is trained on past patient data, but how well it will work on new patients is always worse than how well it looks on the data it already learned from — this gap is called overfitting, and the difference in performance is called optimism. Cross-validation is the standard fix: you repeatedly hide a portion of your patients from the model during training and then test it on those hidden patients, giving you an honest estimate of real-world performance before deploying the model. One critical rule in healthcare data is that you must keep each patient's records together in either training or testing — never split a single patient's claims across both sides — otherwise the model cheats by recognizing patients it has already seen.

What cross-validation and overfitting mean in clinical prediction

Every prediction model — logistic regression for 30-day readmission, a gradient-boosted tree for 1-year mortality, or regularized regression for hospitalization risk — is built by fitting coefficients or splits to observed data. That process is inherently self-referential: the algorithm adjusts its parameters to explain the training labels as well as possible, which means it also memorizes noise specific to those particular patients, specific to that particular calendar window, and specific to that particular database. When you then evaluate the same model on the same data it was trained on, you measure apparent performance — discrimination and calibration that are systematically too good because the model is graded on an exam it has already seen. Cross-validation and bootstrap optimism correction are the two main families of methods that generate an honest, internal estimate of how the model would perform on new patients drawn from the same population — before any external data are touched.

Why apparent performance always flatters: the mechanics of overfitting

Overfitting has two mechanical causes in clinical prediction models. The first is capacity relative to events: a logistic regression with 30 predictors fitted on 150 events (5 events per variable, EPV = 5) has enough degrees of freedom to pick up idiosyncratic patterns in that training sample that will not replicate. The widely cited EPV ≥ 10 rule of thumb warns that models with fewer events per predictor overfit progressively more severely. The second cause is the optimization pressure of fitting: even a correctly specified model with adequate EPV will produce coefficients that are slightly too extreme — the estimated effect of each predictor is pulled toward the direction that best explains the training outcomes, a phenomenon Steyerberg (2001) formalizes as the calibration slope shrinking below 1.0 on held-out data. The net effect is that reported AUC, c-statistic, and calibration metrics computed on training data always overstate how well the model will perform when presented with new patients.

The cross-validation menu

Standard k-fold CV (k = 5 or k = 10 by convention): the analysis dataset is randomly partitioned into k equal folds. In each of the k iterations, one fold serves as the test set and the model is re-trained from scratch on the remaining k − 1 folds; the performance metric (c-statistic, AUC, Brier score) is computed on the held-out fold. The k fold estimates are averaged. The k = 5 convention is well-supported by bias-variance analyses: k = 10 has lower bias but slightly higher variance; k = 2 (simple 50/50 split) is unacceptably noisy; leave-one- out CV is nearly unbiased but computationally expensive and can have high variance.

Stratified k-fold CV: when the outcome event is rare (event rate < 5–10%), pure random splitting of k folds can, by chance, leave one fold with zero events, making the test-set performance undefined. Stratified splitting ensures each fold contains approximately the same event rate as the full dataset. This is the default for any binary prediction model in claims or EHR data with a rare outcome.

Repeated k-fold CV: to reduce the Monte Carlo variance introduced by the random fold assignment, the entire k-fold process is repeated R times (typically R = 5 or 10) with different random seeds and the R × k fold estimates are averaged. Repeated CV is the most stable internal estimate but requires R times as much fitting. It is preferred when n is small (< 500 events) and computational cost is manageable.

Bootstrap optimism correction (Steyerberg/Harrell procedure): fit the model on the full dataset; record the apparent performance metric A. Then draw B bootstrap samples (typically B = 200) from the full dataset with replacement, fit the model to each bootstrap sample, and evaluate that bootstrap-fitted model on the original full dataset. The bootstrap models tend to perform better on their own sample (overfitting) than on the original data; the mean gap across bootstrap replications estimates the optimism O. The corrected estimate is A − O. The bootstrap optimism procedure uses all available data for model fitting in every replicate (no data are withheld) and typically has lower variance than k-fold CV for small samples. In R, the `rms::validate` function implements this directly for logistic and Cox models.

Nested CV (required when hyperparameters are tuned): a fundamental distinction between model selection (choosing which algorithm and hyperparameters to use) and model evaluation (estimating performance of the chosen model) is that performing both with the same CV loop produces optimistic estimates. Varma & Simon (2006) showed that if you select hyperparameters by cross-validation and then report the CV score of the selected model, you have used the test folds to guide selection — the estimate is biased upward. Nested CV uses an outer loop (k-fold) for honest evaluation and an inner loop (inner k-fold or grid search) for hyperparameter selection entirely within the training folds. The outer fold estimate is then unbiased for the full model-selection-plus-fitting pipeline, not just for a fixed algorithm.

RWE-specific data leakage catalog

Data leakage is the most consequential threat to CV validity in claims and EHR studies because it corrupts not just apparent performance but also every cross-validation estimate. Leakage means information that would not be available at the time of prediction — or that violates the patient-independence assumption — flows from the test fold into the model.

Patient-level leakage (claims): when a patient contributes multiple rows (multiple episodes, multiple fill records, multiple encounters), simple random splitting of rows assigns the same patient to both training and test folds. The model implicitly learns patient-level idiosyncratic patterns from the training rows and then "predicts" those patterns in the test rows — a form of identity memorization. The fix is `GroupKFold` (sklearn) or group-stratified splitting by `person_id`, ensuring each patient appears in exactly one fold. This is the most critical leakage mode in claims analysis and the one most often overlooked.

Temporal leakage (prospective deployment): when the model will be deployed prospectively — scoring new patients at time T to predict outcomes that occur after T — a random split that mixes patients indexed in 2018, 2020, and 2022 allows future patients' data to inform training on earlier patients (and vice versa). For prospective deployments, the only valid split is a temporal one: train on the earlier period, validate on the later period. CV with random splits is appropriate only when the model will score a historical population, not when it will be applied forward in time.

Feature leakage (post-index codes): any feature derived from after the prediction time-zero (the index date) is a form of leakage. Common examples: including the treatment's own dispensing codes in a model predicting whether the patient will be treated; using hospitalization codes from the outcome window as predictors; including post-index comorbidity codes in a "baseline" feature matrix. Feature leakage produces models with apparently excellent discrimination that collapse completely at deployment. The solution is strict temporal feature engineering — every predictor must be constructed from data in [index_date − lookback_window, index_date).

Preprocessing leakage: many preprocessing steps — imputation of missing values, standardization of continuous features, feature selection by univariate association — should be fitted only on training data and then applied to test data. If imputation or standardization is fitted on the full dataset before splitting, the test fold implicitly contains information derived from its own values, producing artificially low test-set error. In sklearn, the correct pattern is to wrap all preprocessing in a `Pipeline` so that `fit_transform` is called only on training folds and `transform` (without re-fitting) is called on test folds.

Site leakage (multi-site EHR): when a multi-site dataset is split randomly, training and test sets may contain records from the same site. If the goal is to estimate performance at a new site (leave-one-site-out validation), use `GroupKFold` with site as the grouping variable rather than patient.

Internal versus external validation — the boundary

Cross-validation — including all variants above — is a form of internal validation. It estimates expected performance on new patients drawn from the same source population, same database, same calendar window, and same eligibility criteria as the development cohort. It does not estimate performance in a different health system, a different geographic region, a different era, or a different patient population. Confirming that performance holds in a separate external dataset — a different health plan, a different EHR system, a later calendar period, a different country — is external validation and is covered in this catalog under `prediction-model-validation-recalibration-rwe`. A model that passes internal CV should be thought of as "ready for external validation" rather than "ready for deployment."

Interpreting the output

The concrete example throughout this entry: a logistic regression model for 1-year hospitalization in 500 Medicare patients achieves an apparent c-statistic of 0.85 on the data it was trained on, and 5-fold CV returns fold c-statistics of 0.74, 0.76, 0.78, 0.72, and 0.75, for a CV mean of 0.75 and an optimism of 0.10.

(1) Formal interpretation. The CV estimate of 0.75 approximates the expected discrimination on new patients drawn from the same Medicare population, same database, and same calendar window under the same eligibility criteria. It does NOT estimate performance on a different Medicare Advantage plan, a different geographic region, or a later calendar era — those questions require external validation. The optimism of 0.10 quantifies overfitting: the model memorized patterns worth 0.10 c-statistic points that will not replicate. If any hyperparameters were tuned using the same 5 folds (e.g., a regularization penalty selected by 5-fold CV on the same data), even the 0.75 estimate is biased upward — nested CV is needed to produce an unbiased estimate for the full training pipeline.

(2) Practical interpretation. In plain language: expect about 0.75 discrimination, not 0.85, when this model scores next year's similar patients drawn from this same Medicare system. Whether it performs comparably on patients from a different health system, a different region, or a different calendar era is unknown until an external validation study is completed. The 0.10 optimism gap is a direct signal that the model has overfitted and that shrinkage or regularization should be considered before any external deployment.

Pros, cons, and trade-offs

k-fold CV: - Pros: computationally feasible for most models; k = 5 or 10 has well-characterized bias- variance properties; stratified and grouped variants handle rare outcomes and multi-row patients; nested CV extends to hyperparameter selection. - Cons: each fold trains on only (k − 1)/k of the data, so the estimate applies to a model that is slightly underfit relative to the final model (this is the pessimism of CV); random variation in fold assignment adds variance to the estimate, especially at small n. - When to prefer: moderate to large datasets (n ≥ several hundred events); when bootstrap computation is prohibitive; whenever preprocessing must be validated end-to-end with a Pipeline.

Bootstrap optimism correction: - Pros: fits on the full dataset in each replicate so the corrected estimate applies to the final model, not a slightly underfit k-fold version; lower variance than CV at small sample sizes; directly estimates the shrinkage factor for coefficients. - Cons: computationally expensive (B = 200 full model fits); does not naturally extend to nested hyperparameter selection without additional structure; less transparent to audiences unfamiliar with bootstrap resampling. - When to prefer: small datasets (n < 200 events); when coefficient shrinkage factors are needed; Cox and logistic models implemented in `rms::validate`.

Simple train-test split: - Pros: fast; easy to explain; gives a single unbiased estimate if the split is pre-specified. - Cons: wastes data (typically 20–30% is withheld for testing and not used in training); high variance from a single random partition; particularly unreliable at small n. - When to prefer: only when n is very large (n > 50,000 patients) and an end-to-end evaluation pipeline is needed quickly as a sanity check; never as the primary internal validation in a publication.

When to use

Apply cross-validation or bootstrap optimism correction in every prediction model development study before reporting performance metrics:

• Before presenting a c-statistic, AUC, or calibration metric from a prediction model, always
• When the development sample has fewer than 200 events or EPV < 10, use bootstrap optimism
• When the dataset contains multiple rows per patient (pharmacy fills, encounters, episodes),
• When the model will be deployed prospectively (scoring future patients), use a temporal split
• When tuning hyperparameters (regularization penalty, tree depth, number of features), use
• After internal validation passes, proceed to external validation

When NOT to use — and active misuse patterns

• Reporting apparent (training-set) performance as if it were validation performance: the most
• Random row-level splits when patients have multiple rows: in claims or EHR datasets where
• Using the CV test-fold estimates for model selection and then citing them as unbiased: the
• Applying random CV splits to a prospective deployment question: if the model will score
• Imputing or standardizing features before splitting: fitting an imputer or scaler on the
• Treating a good internal CV estimate as evidence of external validity: a CV c-statistic of

Worked example

Scenario

A logistic regression model is built to predict 1-year hospitalization in a Medicare claims cohort of 500 patients, using 30 baseline features (comorbidity indicators, prior utilization counts). The model is fitted on all 500 patients and the apparent c-statistic — measured on the same data used for fitting — is 0.85. The team then applies 5-fold cross-validation, splitting the 500 patients into 5 groups of 100, training on 400 each time and testing on the 100 held-out patients, to get an honest estimate of how the model will perform on next year's similar Medicare patients.

Dataset

5-fold CV results. Each fold holds out 100 patients as the test set; the model is re-trained on the remaining 400 and the c-statistic is computed on the 100 held-out patients.

Steps

• The apparent c-statistic (model evaluated on its own training data) = 0.85. This is optimistic because the model has already seen these patients.

• Sum the five held-out fold c-statistics: 0.74 + 0.76 + 0.78 + 0.72 + 0.75 = 3.75.

• CV mean c-statistic = 3.75 / 5 = 0.75. This is the honest internal estimate.

• Optimism = apparent c-statistic minus CV mean = 0.85 - 0.75 = 0.10. The model memorized patterns worth 0.10 c-statistic points that will not replicate on new patients.

• The corrected honest estimate (0.75) applies to new Medicare patients from this same database and time window. It does not tell us how the model will perform at a different health system or in a later year — that requires external validation.

Result

Apparent c = 0.85; sum of fold c-stats = 0.74 + 0.76 + 0.78 + 0.72 + 0.75 = 3.75; CV mean c = 3.75 / 5 = 0.75; optimism = 0.85 - 0.75 = 0.10. Honest internal estimate is 0.75.

Runnable example

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import (
    GroupKFold, StratifiedGroupKFold, cross_validate, GridSearchCV
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.metrics import roc_auc_score

# ── Prepare data ──
FEATURE_COLS = [c for c in df.columns if c not in ("person_id", "outcome")]
X = df[FEATURE_COLS].values
y = df["outcome"].values
groups = df["person_id"].values          # CRITICAL: patient-level splitting

# ── Pattern 1: GroupKFold — each patient appears in ONE fold only ──
# Use StratifiedGroupKFold when event rate is < 10% (stratifies by outcome AND groups by patient).
# For event rate >= 10%, plain GroupKFold is fine.
event_rate = y.mean()
if event_rate < 0.10:
    cv_outer = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)
else:
    cv_outer = GroupKFold(n_splits=5)

# ── Pattern 2: Pipeline — all preprocessing fitted only on training folds ──
# SimpleImputer + StandardScaler inside the Pipeline are NEVER fitted on the test fold.
pipe = Pipeline([
    ("impute",  SimpleImputer(strategy="median")),  # fit on train fold only
    ("scale",   StandardScaler()),                  # fit on train fold only
    ("clf",     LogisticRegression(max_iter=500, C=1.0)),
])

# ── Flat 5-fold CV: honest internal estimate ──
results = cross_validate(
    pipe, X, y,
    cv=cv_outer,
    groups=groups,
    scoring="roc_auc",
    return_train_score=True,
)
cv_mean   = results["test_score"].mean()
apparent  = results["train_score"].mean()   # average apparent AUC across folds (proxy)
optimism  = apparent - cv_mean

print(f"CV mean AUC (honest): {cv_mean:.3f}")
print(f"Apparent AUC (train): {apparent:.3f}")
print(f"Optimism:             {optimism:.3f}")

# ── Pattern 3: Nested CV — inner loop selects C, outer loop evaluates honestly ──
# Without nested CV, selecting C by the same folds used for evaluation is biased upward.
param_grid = {"clf__C": [0.01, 0.1, 1.0, 10.0]}
# Inner CV: grouped to respect patient-level integrity inside the training fold.
cv_inner = GroupKFold(n_splits=3)

# GridSearchCV selects the best C using the INNER loop only.
# cross_validate uses the OUTER loop for honest evaluation.
nested_gs = GridSearchCV(
    pipe, param_grid,
    cv=cv_inner,                # inner: hyperparameter selection
    scoring="roc_auc",
    refit=True,
)
nested_results = cross_validate(
    nested_gs, X, y,
    cv=cv_outer,                # outer: honest evaluation of the full pipeline
    groups=groups,
    scoring="roc_auc",
)
nested_cv_mean = nested_results["test_score"].mean()
print(f"Nested CV mean AUC (pipeline + C-selection): {nested_cv_mean:.3f}")
# nested_cv_mean is unbiased for the full train-then-tune-then-predict procedure.
# It is typically lower than flat CV when regularization is tuned on the same folds.

library(rms)
library(rsample)
library(pROC)

# ── 1. Bootstrap optimism correction via rms::validate ──
# lrm fits a logistic regression model; dd stores distribution parameters for rms.
dd <- datadist(dat); options(datadist = "dd")

formula_str <- paste("outcome ~", paste(setdiff(names(dat), c("outcome","person_id")),
                                        collapse = " + "))
fit_lrm <- lrm(as.formula(formula_str), data = dat, x = TRUE, y = TRUE)

# validate() implements bootstrap optimism correction.
# B = 200 bootstrap samples; the default optimism-corrected Dxy (= 2*(C - 0.5)) is reported.
set.seed(42)
val <- validate(fit_lrm, method = "boot", B = 200)

# C-statistic = (Dxy + 1) / 2
apparent_C  <- (val["Dxy", "index.orig"] + 1) / 2
corrected_C <- (val["Dxy", "index.corrected"] + 1) / 2
optimism    <- apparent_C - corrected_C

cat(sprintf("Apparent C (training):  %.3f\n", apparent_C))
cat(sprintf("Corrected C (bootstrap):%.3f\n", corrected_C))
cat(sprintf("Optimism:               %.3f\n", optimism))

# ── 2. Manual k-fold CV with patient-level grouping (rsample::group_vfold_cv) ──
# group_vfold_cv ensures every row of the same patient stays in one fold.
set.seed(42)
folds <- group_vfold_cv(dat, group = "person_id", v = 5)

auc_per_fold <- vapply(folds$splits, function(split) {
  train <- analysis(split)
  test  <- assessment(split)

  # Re-train on the training fold only.
  dd_train <- datadist(train); options(datadist = "dd_train")
  fit_fold <- lrm(as.formula(formula_str), data = train, x = TRUE, y = TRUE)
  pred <- predict(fit_fold, newdata = test, type = "fitted")

  # c-statistic on the held-out fold.
  auc(test$outcome, pred, quiet = TRUE)$auc
}, numeric(1))

cat(sprintf("Fold AUCs: %s\n", paste(round(auc_per_fold, 3), collapse=", ")))
cat(sprintf("CV mean AUC: %.3f\n", mean(auc_per_fold)))
# Apparent minus CV mean = optimism estimate from 5-fold CV.
cat(sprintf("Optimism (apparent %.3f - CV %.3f) = %.3f\n",
            apparent_C, mean(auc_per_fold), apparent_C - mean(auc_per_fold)))