Sample Size, Power, and Precision in RWE

In plain language

Before starting a real-world study, researchers need to show that the data they plan to use contain enough patients and enough disease events to answer their question reliably. Two ways exist to do this: the power approach asks whether the study can reliably detect a meaningful difference if one truly exists, and the precision approach asks how narrow the uncertainty range around the answer will be. In real-world data, these calculations are harder than in a clinical trial because statistical adjustments, the way patients are weighted, and patients leaving the study all shrink the effective information below the raw patient count.

Sample size, power, and precision

is the protocol section that converts a target estimand into a feasibility statement: given the data source, the cohort that can actually be assembled, and the analysis that will be run, can the study either (a) reject a pre-specified null at acceptable type-I/type-II error rates (the power framing) or (b) estimate the effect with a confidence interval narrow enough to inform a decision (the precision framing)? In randomized trials this is a clean closed-form calculation. In real-world evidence it is dominated by features the trial never had: the analysis is adjusted (so variance inflates with confounder–exposure correlation), the comparison is often weighted (so the effective sample size is far below the nominal count), follow-up is administratively censored and competing-risk attrited, and the eligible denominator is eroded by continuous-enrollment and data-completeness requirements. A power calculation that ignores these is not conservative — it is wrong, and it is the single most common reason an "adequately powered" RWE protocol delivers an uninformative interval.

Core conceptual distinction

. Two framings answer two different questions and lead to different numbers. (1) Power (Neyman–Pearson): fix α and a minimum clinically important effect, then solve for the n (or, for time-to-event, the number of events) that achieves 1−β power. This is hypothesis-testing logic — appropriate when the decision is a go/no-go on a pre-specified hypothesis (a safety signal, a non-inferiority margin). (2) Precision (estimation): fix the acceptable confidence-interval half-width on the scale of the estimand (log HR, risk difference, NMB) and solve for the n that delivers it, independent of any null (Greenland 1988, Rothman). This is the framing regulators and HTA bodies increasingly prefer for RWE, because the deliverable is an effect estimate feeding a benefit–risk or cost-effectiveness decision, not a p-value. The estimand drives the math: for a hazard ratio the binding currency is events, not patients (Schoenfeld: required events ≈ (z_{1−α/2}+z_{1−β})² / (φ(1−φ)(log HR)²), where φ is the allocation fraction); for a risk difference or rate it is person-time; for an adjusted estimate it is events per parameter (EPP/EPV), the model-stability constraint (Concato 1995; Peduzzi 1996) that caps how many confounders you can carry before estimates become unstable regardless of total n.

Pros, cons, and trade-offs

. - Power framing vs precision framing: Power is familiar to reviewers and required for any null-hypothesis claim (non-inferiority, safety rule-out). Cost: it hinges on a guessed effect size and silently rewards underpowered studies that "happen" to be significant; a study powered for HR=0.70 says nothing useful if the truth is HR=0.90. Prefer the precision framing whenever the estimand feeds a decision-analytic model (NMB, ICER) or a benefit–risk assessment — report the expected CI half-width, not just power at one alternative. - Events-driven (survival) vs n-driven (binary/continuous) planning: For time-to-event outcomes the calculation must be in events; planning on total n and assuming everyone is followed to the outcome overstates power badly when the outcome is rare or follow-up is short. Cost: events-driven planning requires assumptions about baseline incidence, accrual, and dropout that claims pilot data can supply but registries often cannot. Prefer events-driven (`PROC POWER twosamplesurvival`, `powerSurvEpi`) for any survival endpoint. - Crude vs design-corrected n: A textbook two-sample formula gives the floor. The honest RWE number multiplies it by (i) the variance inflation factor 1/(1−R²) where R² is the multiple correlation of exposure on the confounders (Hsieh 1998) — adjustment for strong confounders can double the required n; (ii) the IPTW/weighting design effect 1 + CV²(weights), which can shrink effective sample size by 30–60% with heavy tails; (iii) competing-risk attrition of events in elderly claims cohorts. Cost: each correction needs a pilot extract. Always design-correct — the uncorrected number is the most dangerous artifact in an RWE protocol.

When to use

. Every comparative RWE protocol (effectiveness, safety, utilization) and every descriptive study whose estimate carries a decision must include a quantitative feasibility statement before data are pulled, calibrated to the actual analysis (adjusted, weighted, time-to-event) and the actual extractable cohort. Use the precision framing when the output is an effect estimate feeding HTA/NMB or benefit–risk; use the power framing for pre-specified hypothesis tests (non-inferiority, safety rule-out, formal subgroup tests). Use EPP/EPV as a hard feasibility gate the moment you decide how many confounders the propensity or outcome model will carry.

When NOT to use — and when it is actively misleading or dangerous

. A crude unadjusted power calculation pasted into an RWE protocol is worse than none: it manufactures false confidence and is routinely rejected on review. It is actively misleading when (1) the analysis is weighted but the calculation uses the nominal n — the realized CI will be far wider than promised because the design effect was ignored; (2) the endpoint is time-to-event with meaningful competing risks but planning assumed the cause-specific events accrue at the marginal rate — competing death in an elderly comparator arm steals the very events you powered on, and the theft is differential by exposure; (3) the calculation is used as a post hoc "observed power" rationalization of a null result (observed power is a deterministic transform of the p-value and tells you nothing — report the CI instead); (4) immortal-time or look-back requirements that erode the denominator are omitted, so the "eligible n" in the protocol never materializes after enrollment rules are applied. For any well-powered safety screen across hundreds of outcomes, single-comparison power is also misleading without an explicit multiplicity / false-discovery plan.

Data-source operational depth

. - Claims (FFS vs Medicare Advantage): The eligible denominator is not the enrolled population — it is the subset with continuous medical + pharmacy enrollment spanning the full washout and look-back, which can drop 40–70% of members and must be modeled in the feasibility calc, not discovered after the fact. MA-only person-time lacks adjudicable FFS claims, so events and exposures are undercounted; if the database mixes FFS and MA, power computed on the headcount is inflated — restrict the denominator to FFS-observable person-time before computing expected events. Differential competing risks (death, disenrollment) by exposure in elderly claims attrite cause-specific events asymmetrically; plan events with a competing-risk-adjusted cumulative incidence, not 1−KM. Claims pilot extracts are the right place to estimate comparator-arm incidence and the weight distribution that drives the IPTW design effect. - EHR: Capture is encounter-driven, so the observed event rate understates the true rate (silent outcomes outside the system) and follow-up is fragmented by leakage to other providers — both reduce realized events below the planning assumption. Use linked claims, where available, to anchor the incidence estimate; otherwise treat the EHR-based rate as a lower bound and inflate the planned n. - Registry: Often strong for adjudicated outcomes and severity (so the per-event information is high) but small and selectively enrolled, making the precision framing essential — a registry rarely powers a hypothesis test but can deliver a usefully narrow interval for a moderate effect. Account for registry incompleteness in the denominator. - Linked claims–EHR–vital records: The ideal substrate for accurate event and incidence estimates, but the linkable subset is a selected denominator; compute feasibility on the linked population, not the source frame, and reconcile date discrepancies before defining person-time.

Worked claims example

Question: power/precision for incident heart failure with second-generation sulfonylurea vs DPP-4 inhibitor, IPTW-adjusted Cox, in a commercial + Medicare FFS database; minimum clinically important HR = 0.80, two-sided α=0.05, target 90% power. (1) Denominator erosion: start from 4.0M diabetes members; require ≥18y, ≥2 diabetes Dx, and 365 days continuous A/B/D (or commercial medical+pharmacy) enrollment before the first qualifying fill — this leaves ~620k; the new-user washout (no prior sulfonylurea/DPP-4 fill in 365d) and FFS-observable restriction (exclude MA-only person-time) leave ~180k initiators, roughly 45% sulfonylurea / 55% DPP-4. (2) Events, not patients: with a comparator-arm 3-year cumulative HF incidence of ~6% and a competing mortality of ~9% that differentially attrites events, expected cause-specific HF events ≈ 0.055 × 180k ≈ 9,900 over 3 years of administratively censored follow-up — but Schoenfeld requires only events ≈ (1.96+1.28)² / (0.5·0.5·(ln 0.80)²) ≈ 10.51 / (0.25·0.0497) ≈ 845 events for 90% power at HR=0.80. (3) Design correction: multiply by the IPTW design effect 1 + CV²(weights); with a stabilized-weight CV ≈ 0.6 that is ×1.36 → ~1,150 effective events needed, and by the confounder variance-inflation 1/(1−R²) with R²≈0.15 → ×1.18 → ~1,360 events. The ~9,900 accrued events comfortably exceed 1,360, so the study is well powered for HR=0.80 and, more usefully, the precision statement is the deliverable: with ~9,900 events the expected 95% CI half-width on ln HR is z·√(1/(φ(1−φ)·E_eff)) ≈ 1.96·√(1/(0.25·7,300)) ≈ 0.046, i.e., an HR estimate of 0.80 would carry a 95% CI of roughly 0.76–0.84 — narrow enough to inform a benefit–risk decision. (4) If underpowered (e.g., the comparator were rarely used and events fell below ~1,360): extend the follow-up window, broaden the comparator to a clinically interchangeable class, switch from a hypothesis test to an explicit precision target, or accept a wider but still decision-relevant CI rather than overstating power. Also confirm EPP: the IPTW Cox + any residual adjustment must carry no more parameters than ~events/10 ≈ 130 — never a binding constraint here, but decisive in a rare-outcome registry study.

Interpreting the output

Consider the claims example above: a Schoenfeld calculation for an IPTW-adjusted Cox model targeting HR = 0.80 with 90% power. After applying the IPTW design-effect multiplier (≈ ×1.36) and the confounder inflation factor (≈ ×1.18), the study requires approximately 1,360 effective events. The database yields ≈ 9,900 accrued events, far exceeding that threshold. The primary deliverable is then the precision statement: with ≈ 9,900 events, an observed HR of 0.80 would carry a 95% CI of roughly 0.76–0.84.

(1) Formal statistical interpretation. The power calculation is a pre-data design property: it describes the long-run probability — across hypothetical replications — that the study would reject the null if the true HR were exactly 0.80. Post-hoc "observed power" is a deterministic function of the p-value and adds no information beyond the CI; report the CI, not observed power, for null results. The IPTW design effect inflates the required event count because weighted observations are not independent; ignoring it produces an event count that is nominally adequate but statistically underpowered.

(2) Practical interpretation for a decision-maker. With ≈ 9,900 events, the study can estimate the hazard ratio with a 95% CI spanning roughly ±0.04 on the HR scale — tight enough to distinguish HR 0.80 from HR 0.76 and from HR 0.84. That precision supports a benefit–risk or formulary decision. The power target (90%) is a pre-commitment about acceptable design risk, not a statement about the specific study's conclusion.

Worked example

Scenario

A registry study compares two diabetes medications, sulfonylureas (SU) and DPP-4 inhibitors, on the risk of hospitalization for heart failure over three years. The source file uses comparator-arm (DPP-4) incidence of 6.0% and study-arm (SU) incidence of 4.8%, giving a true risk difference of 1.2 percentage points. Equal numbers of patients enter each arm. How wide will the confidence interval be at different total study sizes, and when does the study become precise enough to inform a decision?

Dataset

Planning table: total study size, arm size (half each), and the resulting 95% confidence interval half-width around the 1.2 pp risk difference.

Steps

• The true risk difference we are trying to estimate is 6.0% minus 4.8% equals 1.2 percentage points.

• For each total N, split patients evenly: 100 per arm at N=200, 500 per arm at N=1000, and so on.

• The standard error (SE) of a risk difference is the square root of p1(1-p1)/(arm size) plus p2(1-p2)/(arm size). At N=1000 this is sqrt(0.0480.952/500 + 0.0600.940/500) = sqrt(0.0000913 + 0.0001128) = 0.0143.

• Multiply SE by 1.96 to get the 95% confidence interval half-width: 1.96 * 0.0143 = 0.028, or 2.8 percentage points.

• At N=200, the half-width is 6.3 pp, so the CI nearly straddles zero and the result is uninformative; at N=2000 the half-width drops to 2.0 pp, which may be narrow enough to influence a prescribing decision.

• In a real-world claims study (the source file scales this to 180k patients) the effective sample size is smaller than the headcount because weighting and adjustment consume information, so the honest CI is wider than this simple table shows.

Result

N=1000 gives a 95% CI of roughly -1.6 to +4.0 percentage points (half-width 2.8 pp). N=2000 tightens this to roughly -0.8 to +3.2 pp (half-width 2.0 pp). Whether 2 pp is narrow enough depends on what decision the estimate feeds; in many health technology assessments, a CI that wide still spans both clinically relevant and clinically unimportant differences, which is why the source file targets roughly 9,900 events in its large claims cohort to achieve a half-width of about 4.6% on the log-hazard-ratio scale.

Runnable example

import numpy as np
from scipy.stats import norm
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

def events_for_hr(hr, phi=0.5, alpha=0.05, power=0.90):
    """Schoenfeld required number of EVENTS (not patients) for a two-arm HR.
    phi = fraction allocated to the treated arm."""
    z = norm.ppf(1 - alpha / 2) + norm.ppf(power)
    return z**2 / (phi * (1 - phi) * (np.log(hr)) ** 2)

def n_for_risk_diff(p_comparator, p_study, ratio=1.0, alpha=0.05, power=0.90):
    """Total n for a two-proportion comparison (binary outcome), arm-balanced by `ratio`."""
    h = proportion_effectsize(p_study, p_comparator)         # Cohen's h on arcsine scale
    n1 = NormalIndPower().solve_power(effect_size=abs(h), alpha=alpha,
                                      power=power, ratio=ratio, alternative="two-sided")
    return np.ceil(n1) * (1 + ratio)

def design_correct(n_or_events, weight_cv=0.0, conf_r2=0.0):
    """Inflate crude n/events for the IPTW design effect (1 + CV^2 of weights)
    and the confounder variance-inflation factor 1/(1 - R^2) (Hsieh 1998)."""
    deff = 1.0 + weight_cv**2
    vif  = 1.0 / (1.0 - conf_r2)
    return np.ceil(n_or_events * deff * vif)

def epp_max_params(n_events, epp=10):
    """Events-per-parameter feasibility cap for the adjusted (outcome) model."""
    return int(n_events // epp)

def ci_halfwidth_loghr(expected_events, phi=0.5, alpha=0.05):
    """Precision framing: expected 95% CI half-width on the log-HR scale given accrued events.
    Var(logHR) ~ 1 / (phi*(1-phi)*E). Exponentiate point +/- this for the HR-scale interval."""
    z = norm.ppf(1 - alpha / 2)
    return z * np.sqrt(1.0 / (phi * (1 - phi) * expected_events))

# --- planning the sulfonylurea vs DPP-4 HF example ---
crude = events_for_hr(hr=0.80, phi=0.45, power=0.90)          # ~ required events at HR 0.80
needed = design_correct(crude, weight_cv=0.60, conf_r2=0.15)  # IPTW + confounding inflation
print("crude events:", round(crude), " design-corrected:", needed)
print("max model params at EPV>=10:", epp_max_params(9900))
print("expected log-HR CI half-width:", round(ci_halfwidth_loghr(9900, phi=0.45), 3))

library(pwr)
library(epiR)
library(powerSurvEpi)

## 1. Events-driven survival planning (Schoenfeld) for a target HR.
events_for_hr <- function(hr, phi = 0.5, alpha = 0.05, power = 0.90) {
  z <- qnorm(1 - alpha / 2) + qnorm(power)
  z^2 / (phi * (1 - phi) * (log(hr))^2)
}

## 2. Cohort-study sample size for a binary outcome / risk comparison.
n_binary <- epiR::epi.sscohortc(
  irexp1 = 0.048, irexp0 = 0.060,        # incidence in study vs comparator arm
  n = NA, power = 0.90, r = 0.45 / 0.55,  # allocation ratio study:comparator
  conf.level = 0.95
)$n.total

## 3. Design-correct crude n/events: IPTW design effect x confounder VIF.
design_correct <- function(x, weight_cv = 0, conf_r2 = 0) {
  ceiling(x * (1 + weight_cv^2) * (1 / (1 - conf_r2)))
}

## 4. Precision framing: expected 95% CI half-width on the log-HR scale.
ci_halfwidth_loghr <- function(events, phi = 0.5, alpha = 0.05) {
  qnorm(1 - alpha / 2) * sqrt(1 / (phi * (1 - phi) * events))
}

crude  <- events_for_hr(0.80, phi = 0.45, power = 0.90)
needed <- design_correct(crude, weight_cv = 0.60, conf_r2 = 0.15)
cat("crude events:", round(crude), " design-corrected:", needed, "\n")
cat("binary-outcome total n:", round(n_binary), "\n")
cat("expected log-HR CI half-width:", round(ci_halfwidth_loghr(9900, 0.45), 3), "\n")

## 5. (optional) powerSurvEpi for power given a fixed number of subjects + follow-up.
## ssizeCT.default(power = 0.90, k = 0.45/0.55, pE = 0.048, pC = 0.060, RR = 0.80, alpha = 0.05)