Two-Phase Validation Sampling Design

In plain language

Two-phase validation sampling means you do a large cheap database study first, then carefully choose a smaller subset for expensive truth-finding. The subset is not just a random audit; it is designed so chart review or linkage can estimate the exact error or missing-confounder parameters needed to correct the main study.

Two-phase validation sampling

is the design backbone behind many credible RWE bias corrections. Phase 1 is the full cohort or source population: claims, EHR, registry, or linked data provide treatment, outcome proxies, baseline covariates, enrollment, and follow-up for everyone. Phase 2 is a sampled subset where a costly "better measurement" is obtained: chart-adjudicated outcomes, true exposure status, smoking/BMI/lab/severity covariates, registry stage, death-certificate details, or manual abstraction of endpoints. The point is not to review a convenient set of charts. The point is to sample enough of the right patients so the validation data identify the sensitivity/specificity, calibration equation, or measurement-error model needed to correct the phase-1 analysis.

Core design distinction

A validation substudy is only useful if its sampling design matches the parameter needed for correction. Sampling only algorithm-positive patients estimates positive predictive value (PPV), not sensitivity. Sampling only events estimates chart-confirmed case composition, not false negatives. Sampling only patients with available EHR charts estimates accuracy among chart-available patients, not the whole claims cohort. The design must specify: the phase-1 population, strata used for sampling, phase-2 sampling fractions, gold-standard measurement, linkage/chart-availability rules, analysis weights, and how uncertainty from the validation phase enters the final effect estimate.

Pros, cons, and trade-offs

- vs simple random chart review: Stratified two-phase sampling oversamples informative cells, such as algorithm positives, algorithm negatives, exposure arms, PS tails, rare outcomes, or discordant data patterns. Cost: analysts must retain sampling probabilities and use inverse-probability, likelihood, mean-score, or calibration-weighted analysis. - vs complete validation: Two-phase designs make expensive validation feasible and can be nearly as efficient when strata are chosen well. Cost: sparse strata, nonresponse, and unavailable charts can compromise identifiability. - vs external published validation parameters: Internal two-phase validation measures the algorithm or confounder in the same source population, payer mix, calendar era, and coding system. Cost: it takes time, chart-access agreements, and adjudication infrastructure. - vs ad hoc "10% sample" audits: A fixed-percentage audit often wastes reviews on low-information records. A planned two-phase design targets the phase-2 sample to the bias parameter or regression coefficient that drives the decision.

When to use

Use a two-phase validation design when phase-1 RWD are large enough for the target analysis but an expensive or unavailable variable threatens validity. Typical triggers: claims endpoint algorithms with unknown sensitivity, EHR exposure fields with suspected error, registry linkage with selected capture, missing severity/lifestyle covariates, NLP phenotypes requiring manual adjudication, or a regulatory/HTA submission where residual bias must be quantified rather than described. Pre-specify the validation sampling plan before adjudication begins. For a misclassified outcome, sample across algorithm-positive and algorithm-negative strata and across exposure arms when differential misclassification is plausible. For unmeasured confounding, sample across treatment arms and propensity-score tails so the validation subset can learn the missing-confounder distribution where it matters.

When NOT to use -- and when it is actively misleading or dangerous

- The "gold standard" is not actually better than phase 1. Chart review cannot validate events that occur outside the health system unless outside records are obtained; registry truth may lag or miss community cases; NLP labels may inherit documentation bias. - The validation frame is selected after observing charts. Dropping unavailable charts without modeling availability converts validation into a convenience sample and can bias sensitivity/specificity or calibration estimates. - Sampling does not identify the needed parameter. PPV from reviewed positives cannot correct true incidence without information on false negatives. Sensitivity and specificity need data on true cases and true non-cases, or a design and model that can recover them. - Differential error is plausible but ignored. If outcome capture differs by treatment arm, site, payer, or surveillance intensity, pooled validation parameters can move the corrected estimate in the wrong direction. - Sampling probabilities are lost. Without the phase-2 selection probabilities and nonresponse information, a weighted or likelihood-based correction cannot be audited.

Data-source operational depth

- Claims: Phase 1 usually has complete exposure, enrollment, and coded events for FFS or commercial medical+pharmacy members. Validate only in periods and payer segments where a code-negative record is interpretable. Medicare Advantage encounter incompleteness can turn algorithm-negative into "not observed"; do not estimate false-negative rates from incomplete capture. - EHR: Phase 2 can abstract notes, labs, vitals, imaging, smoking, BMI, and severity. Sampling should account for site, visit intensity, chart availability, and outside-care leakage. If only high-utilization patients have rich notes, chart review can overstate sensitivity. - Registry: Registry linkage can supply adjudicated diagnosis, stage, recurrence, mortality, or device details, but registry inclusion and linkage success are themselves selection processes. Sample or weight by linkability when applying validation parameters to the full cohort. - Linked data: Linked phase-2 validation is powerful but needs an explicit linkability diagram: who was eligible for linkage, who matched, who had enough source data for adjudication, and how those groups differ from the full phase-1 cohort.

Worked RWE example

A Medicare FFS study compares Drug A and Drug B for hospitalized stroke. Phase 1 defines stroke using an inpatient ICD-10 algorithm for 80,000 new users. The team needs sensitivity and specificity, not only PPV, because the effect estimate is a risk ratio. They create phase-1 strata by treatment arm, algorithm status, age group, site, and high/low baseline stroke risk. They oversample algorithm-positive records for PPV and enough algorithm-negative records to detect false negatives, with separate sampling fractions by arm. Abstractors adjudicate stroke from hospital charts while blinded to treatment. The analysis uses phase-2 sampling weights to estimate arm-specific sensitivity and specificity and propagates those estimates into a probabilistic misclassification correction. The substudy is defensible because it was sampled from the same FFS-complete cohort and was designed to identify the parameters needed by the correction.

Worked example

Scenario

A claims endpoint algorithm identifies possible hospitalized stroke, but the team needs chart-adjudicated sensitivity and specificity to correct an effect estimate. They design a phase-2 chart review from the phase-1 cohort.

Dataset

Example phase-2 allocation by algorithm status and treatment arm

Steps

• Define the phase-1 cohort, algorithm status, treatment arm, and risk strata before chart review.

• Choose phase-2 sampling fractions that oversample algorithm positives and high-risk algorithm negatives.

• Retain each subject's sampling probability and chart-availability status.

• Estimate PPV, sensitivity, and specificity with validation weights, stratified by treatment arm if surveillance differs.

• Feed those estimates into a probabilistic misclassification correction or a likelihood-based outcome model.

Result

The design identifies more than PPV: it supplies information on false negatives and allows arm-specific correction because both algorithm-positive and algorithm-negative records were reviewed within each arm.

Runnable example

import numpy as np
import pandas as pd

def draw_validation_sample(phase1, n_per_stratum, seed=42):
    rng = np.random.default_rng(seed)
    strata = ["treatment", "algorithm_status", "risk_stratum"]
    out = []
    for key, g in phase1.groupby(strata, dropna=False):
        target = min(len(g), n_per_stratum.get(key, n_per_stratum.get("default", 50)))
        picked = g.sample(n=target, random_state=int(rng.integers(0, 1_000_000)))
        picked = picked.copy()
        picked["phase1_stratum_n"] = len(g)
        picked["phase2_sample_n"] = target
        picked["phase2_sampling_fraction"] = target / len(g)
        picked["validation_weight"] = len(g) / target
        out.append(picked)
    return pd.concat(out, ignore_index=True)

# Example: n_per_stratum can key exact tuple strata or use "default".
# validation_frame = draw_validation_sample(phase1, {"default": 75})

library(dplyr)

draw_validation_sample <- function(phase1, n_default = 50, seed = 42) {
  set.seed(seed)
  phase1 %>%
    group_by(treatment, algorithm_status, risk_stratum) %>%
    group_modify(function(.x, .y) {
      n1 <- nrow(.x)
      n2 <- min(n1, n_default)
      picked <- slice_sample(.x, n = n2)
      mutate(picked,
             phase1_stratum_n = n1,
             phase2_sample_n = n2,
             phase2_sampling_fraction = n2 / n1,
             validation_weight = n1 / n2)
    }) %>%
    ungroup()
}