Two-Phase Validation Sampling Design

Two-phase validation sampling

is the design backbone behind many credible RWE bias corrections. Phase 1 is the full cohort or source population: claims, EHR, registry, or linked data provide treatment, outcome proxies, baseline covariates, enrollment, and follow-up for everyone. Phase 2 is a sampled subset where a costly "better measurement" is obtained: chart-adjudicated outcomes, true exposure status, smoking/BMI/lab/severity covariates, registry stage, death-certificate details, or manual abstraction of endpoints. The point is not to review a convenient set of charts. The point is to sample enough of the right patients so the validation data identify the sensitivity/specificity, calibration equation, or measurement-error model needed to correct the phase-1 analysis.

Core design distinction

A validation substudy is only useful if its sampling design matches the parameter needed for correction. Sampling only algorithm-positive patients estimates positive predictive value (PPV), not sensitivity. Sampling only events estimates chart-confirmed case composition, not false negatives. Sampling only patients with available EHR charts estimates accuracy among chart-available patients, not the whole claims cohort. The design must specify: the phase-1 population, strata used for sampling, phase-2 sampling fractions, gold-standard measurement, linkage/chart-availability rules, analysis weights, and how uncertainty from the validation phase enters the final effect estimate.

Pros, cons, and trade-offs

• vs simple random chart review: Stratified two-phase sampling oversamples informative cells, such as algorithm positives, algorithm negatives, exposure arms, PS tails, rare outcomes, or discordant data patterns. Cost: analysts must retain sampling probabilities and use inverse-probability, likelihood, mean-score, or calibration-weighted analysis.
• vs complete validation: Two-phase designs make expensive validation feasible and can be nearly as efficient when strata are chosen well. Cost: sparse strata, nonresponse, and unavailable charts can compromise identifiability.
• vs external published validation parameters: Internal two-phase validation measures the algorithm or confounder in the same source population, payer mix, calendar era, and coding system. Cost: it takes time, chart-access agreements, and adjudication infrastructure.
• vs ad hoc "10% sample" audits: A fixed-percentage audit often wastes reviews on low-information records. A planned two-phase design targets the phase-2 sample to the bias parameter or regression coefficient that drives the decision.

When NOT to use -- and when it is actively misleading or dangerous

• The "gold standard" is not actually better than phase 1. Chart review cannot validate events that occur outside the health system unless outside records are obtained; registry truth may lag or miss community cases; NLP labels may inherit documentation bias.
• The validation frame is selected after observing charts. Dropping unavailable charts without modeling availability converts validation into a convenience sample and can bias sensitivity/specificity or calibration estimates.
• Sampling does not identify the needed parameter. PPV from reviewed positives cannot correct true incidence without information on false negatives. Sensitivity and specificity need data on true cases and true non-cases, or a design and model that can recover them.
• Differential error is plausible but ignored. If outcome capture differs by treatment arm, site, payer, or surveillance intensity, pooled validation parameters can move the corrected estimate in the wrong direction.
• Sampling probabilities are lost. Without the phase-2 selection probabilities and nonresponse information, a weighted or likelihood-based correction cannot be audited.

Data-source operational depth

• Claims: Phase 1 usually has complete exposure, enrollment, and coded events for FFS or commercial medical+pharmacy members. Validate only in periods and payer segments where a code-negative record is interpretable. Medicare Advantage encounter incompleteness can turn algorithm-negative into "not observed"; do not estimate false-negative rates from incomplete capture.
• EHR: Phase 2 can abstract notes, labs, vitals, imaging, smoking, BMI, and severity. Sampling should account for site, visit intensity, chart availability, and outside-care leakage. If only high-utilization patients have rich notes, chart review can overstate sensitivity.
• Registry: Registry linkage can supply adjudicated diagnosis, stage, recurrence, mortality, or device details, but registry inclusion and linkage success are themselves selection processes. Sample or weight by linkability when applying validation parameters to the full cohort.
• Linked data: Linked phase-2 validation is powerful but needs an explicit linkability diagram: who was eligible for linkage, who matched, who had enough source data for adjudication, and how those groups differ from the full phase-1 cohort.

Worked RWE example

A Medicare FFS study compares Drug A and Drug B for hospitalized stroke. Phase 1 defines stroke using an inpatient ICD-10 algorithm for 80,000 new users. The team needs sensitivity and specificity, not only PPV, because the effect estimate is a risk ratio. They create phase-1 strata by treatment arm, algorithm status, age group, site, and high/low baseline stroke risk. They oversample algorithm-positive records for PPV and enough algorithm-negative records to detect false negatives, with separate sampling fractions by arm. Abstractors adjudicate stroke from hospital charts while blinded to treatment. The analysis uses phase-2 sampling weights to estimate arm-specific sensitivity and specificity and propagates those estimates into a probabilistic misclassification correction. The substudy is defensible because it was sampled from the same FFS-complete cohort and was designed to identify the parameters needed by the correction.

Decision diagram

flowchart TD
  P1[Phase 1 full RWE cohort<br/>cheap data on all N] --> Strat[Define validation strata<br/>arm, algorithm status, risk, site]
  Strat --> Sample[Sample phase 2 records<br/>known probabilities]
  Sample --> Truth[Gold-standard validation<br/>chart, registry, enriched EHR]
  Truth --> Params[Estimate error or calibration parameters<br/>with validation weights]
  Params --> Correct[Correct phase-1 effect estimate<br/>and propagate uncertainty]

Worked example

Scenario

A claims endpoint algorithm identifies possible hospitalized stroke, but the team needs chart-adjudicated sensitivity and specificity to correct an effect estimate. They design a phase-2 chart review from the phase-1 cohort.

Dataset

Example phase-2 allocation by algorithm status and treatment arm

Steps

Result

The design identifies more than PPV: it supplies information on false negatives and allows arm-specific correction because both algorithm-positive and algorithm-negative records were reviewed within each arm.

Trade-offs

Runnable example

import numpy as np
import pandas as pd

def draw_validation_sample(phase1, n_per_stratum, seed=42):
    rng = np.random.default_rng(seed)
    strata = ["treatment", "algorithm_status", "risk_stratum"]
    out = []
    for key, g in phase1.groupby(strata, dropna=False):
        target = min(len(g), n_per_stratum.get(key, n_per_stratum.get("default", 50)))
        picked = g.sample(n=target, random_state=int(rng.integers(0, 1_000_000)))
        picked = picked.copy()
        picked["phase1_stratum_n"] = len(g)
        picked["phase2_sample_n"] = target
        picked["phase2_sampling_fraction"] = target / len(g)
        picked["validation_weight"] = len(g) / target
        out.append(picked)
    return pd.concat(out, ignore_index=True)

# Example: n_per_stratum can key exact tuple strata or use "default".
# validation_frame = draw_validation_sample(phase1, {"default": 75})

library(dplyr)

draw_validation_sample <- function(phase1, n_default = 50, seed = 42) {
  set.seed(seed)
  phase1 %>%
    group_by(treatment, algorithm_status, risk_stratum) %>%
    group_modify(function(.x, .y) {
      n1 <- nrow(.x)
      n2 <- min(n1, n_default)
      picked <- slice_sample(.x, n = n2)
      mutate(picked,
             phase1_stratum_n = n1,
             phase2_sample_n = n2,
             phase2_sampling_fraction = n2 / n1,
             validation_weight = n1 / n2)
    }) %>%
    ungroup()
}

proc sort data=work.phase1;
  by treatment algorithm_status risk_stratum;
run;

proc surveyselect data=work.phase1
  out=work.validation_frame
  method=srs
  sampsize=50
  seed=42
  noprint;
  strata treatment algorithm_status risk_stratum;
run;

data work.validation_frame;
  set work.validation_frame;
  phase2_sampling_fraction = SelectionProb;
  validation_weight = 1 / SelectionProb;
run;

Citations

1. [1]Breslow NE, Chatterjee N. Design and analysis of two-phase studies with binary outcome applied to Wilms tumour prognosis. Journal of the Royal Statistical Society: Series C (Applied Statistics). 1999;48(4):457-468.
2. [2]Amorim G, Tao R, Lotspeich SC, Shaw PA, Lumley T, Shepherd BE. Two-phase sampling designs for data validation in settings with covariate measurement error and continuous outcome. Journal of the Royal Statistical Society: Series A. 2021;184(4):1368-1389.

1. [3]Shepherd BE, Han K, Chen T, Bian A, et al. Multiwave validation sampling for error-prone electronic health records. Biometrics. 2023;79(3):2649-2663.

1. [4]Stürmer T, Schneeweiss S, Avorn J, Glynn RJ. Adjusting effect estimates for unmeasured confounding with validation data using propensity score calibration. American Journal of Epidemiology. 2005;162(3):279-289.