Survey Weights and Complex Sampling

In plain language

When a health survey deliberately recruits more people from some groups than others — so those groups are reliably measured — each respondent gets a sampling weight that tells you how many real people in the country that one person represents. You multiply each respondent's answer by their weight when computing any national average or count, so the final number reflects the whole population rather than the mix of people who happened to be recruited. Skipping the weights and computing a plain average gives the wrong answer whenever the groups that were oversampled differ from the rest of the country on the outcome you care about.

Survey weights and complex sampling

are the design-based machinery that converts a non-self-weighting probability sample into estimates for a defined finite target population. Three design features must be carried through every analysis. (1) The sampling weight for a respondent is (roughly) the inverse of their probability of selection, often further adjusted for nonresponse and post-stratified/calibrated to known population totals; it tells you how many population members each respondent "stands for." (2) Stratification (e.g., by region, age, race/ethnicity) means the sample was drawn within fixed subpopulations, which reduces variance and must be acknowledged or standard errors are overstated. (3) Clustering (multistage selection of PSUs such as counties, then households, then persons) means observations within a PSU are correlated, which inflates variance via the design effect (DEFF) and must be acknowledged or standard errors are badly understated. In RWE/HEOR this is the engine behind nationally representative estimates from NHANES, MEPS, NHIS, NIS/HCUP, and MCBS — the source of population denominators, prevalence, and per-capita cost and utilization figures that feed budget-impact and cost-of-illness models.

Core conceptual distinction

Two distinctions are doing the work. First, design-based vs model-based inference: survey-weighted (Horvitz–Thompson / design-based) estimation targets a finite-population quantity (the mean/total/ proportion that exists in the actual population) and gets its variance from the sampling design, not from an assumed data- generating model. Second — and this is the one analysts get wrong — when weighting actually helps. For descriptive population estimands (national mean cost, prevalence, total person-years), weights are mandatory: an unweighted mean estimates the mean of the sample, which is not the population mean when selection probabilities differ. For causal regression estimands, weighting is more subtle: Solon, Haider & Wooldridge (2015) show that if your conditional model is correctly specified and selection is exogenous to the error, sampling weights add variance without removing bias; weights are warranted when the model is misspecified and you want a population-averaged effect, when selection is on the outcome (endogenous sampling), or when heterogeneous effects make the population-weighted average the estimand of interest. A separate, frequently conflated object is the inverse-probability-of-treatment weight (IPTW) from propensity scores: IPTW and sampling weights share Horvitz–Thompson arithmetic but answer different questions (IPTW balances confounders for a causal contrast; sampling weights restore population representativeness). The genuine RWE bridge between the two is the inverse-probability-of-sampling weight (IPSW) used to transport a trial or cohort effect to a target population (Cole & Stuart 2010) — that is a survey-weighting idea applied to generalizability, not the same thing as IPTW.

Pros, cons, and trade-offs

- vs naive unweighted analysis of survey data: Design-based estimation yields the right population point estimate and, just as importantly, the right variance (Taylor linearization or replicate weights). Cost: estimators are more complex, weighted estimates have higher variance than a hypothetical equal-probability sample of the same size (the price of unequal selection, quantified by DEFF/UWE), and subgroup precision can be poor. Always prefer design-based estimation for any inferential claim about the population a survey was designed to represent. - vs treating survey weights as frequency/precision weights in an off-the-shelf GLM: Frequency-weight software returns the correct weighted point estimate but model-based standard errors that ignore stratification and clustering — almost always too small (clustering dominates), producing falsely narrow CIs and inflated type-I error. Prefer true survey procedures (`svyglm`, `PROC SURVEYREG`/`SURVEYLOGISTIC`, `samplics`) that consume the full design object. - vs model-based / multilevel modeling of the same clustered data: A mixed model can target a similar variance structure and is more efficient under correct specification, but it is model-dependent and targets a different (often cluster-conditional) estimand. Prefer design-based estimation when the deliverable is a population total/mean for an agency or payer and robustness to model misspecification matters more than efficiency.

When to use

The data come from a complex probability survey (NHANES, MEPS, NHIS, MCBS, NIS/HCUP, BRFSS) or any sample drawn with unequal selection probabilities, stratification, or clustering, and the estimand is a finite-population quantity (prevalence, national totals, per-capita cost/utilization, population-representative regression coefficients). Use the survey-provided weight, strata, and PSU variables exactly as documented; use the longitudinal weight (not the annual cross-sectional weight) for panel estimands such as two-year MEPS expenditures; and use replicate weights (BRR/jackknife) when the survey ships them instead of, or alongside, Taylor-linearization design variables. The same machinery applies when you construct IPSW to transport a study estimate to a named target population.

When NOT to use — and when it is actively misleading or dangerous

- Pure administrative claims/EHR cohorts carry no survey weights. Applying invented "weights" to a convenience cohort does not make it representative; the real problem there is generalizability/transportability, which is IPSW/standardization territory, not design-based survey estimation. Fabricating weights manufactures false confidence in a non-probability sample. - Subgroup ("domain") analysis by filtering the data frame. This is the single most dangerous error: dropping rows for a subpopulation deletes PSUs/strata, corrupts the degrees of freedom and the between-PSU variance, and yields wrong standard errors. Domain estimation must keep the full design and subset the design object (R `subset()` on the design, SAS `DOMAIN` statement). Filtering first can shift a CI enough to flip a significance call. - Causal-effect estimation where the conditional model is correctly specified and selection is exogenous. Per Solon et al., weighting here only inflates variance — report both, and do not weight reflexively. - Ignoring the design entirely. Treating a clustered, stratified sample as i.i.d. understates variance (often 1.5–3× DEFF), turning noise into "significant" findings — the canonical reason payer/HTA reviewers reject survey-based estimates.

Data-source operational depth

- NHANES/MEPS/NHIS/MCBS (designed surveys): Weights, strata (e.g., `SDMVSTRA`), and PSU (`SDMVPSU`) are documented and must be used verbatim. Failure modes: (a) using the wrong weight when combining survey cycles (NHANES requires constructing multi-cycle weights, e.g., dividing the 2-year weight when pooling); (b) using MEC/exam weights for interview-only variables, or interview weights for lab variables — they have different nonresponse adjustments; (c) using a cross-sectional weight for a longitudinal estimand (MEPS panel costs need the longitudinal weight that accounts for wave attrition); (d) domain subsetting by filtering rather than the design object. - NIS/HCUP (hospital-discharge survey): The unit is the discharge, not the patient, and HCUP redesigned its weights (pre/post-2012 trend weights). Failure modes: treating discharges as persons; ignoring the hospital cluster and discharge weight, which makes national-estimate CIs far too narrow. - Survey linked to claims/mortality (e.g., MEPS- or MCBS-linked Medicare, NHANES Linked Mortality): Linkage gives true longitudinal outcomes but breaks representativeness because only the consentable/linkable subset is retained; differential consent by age/race/insurance must be addressed with linkage-eligibility-adjusted weights, or the survey weight no longer maps to the population. - Registries and convenience EHR samples: Design weights rarely exist. At most, post-stratification/raking weights to external population margins can be constructed, but these correct only for the margins used and cannot fix selection on unmeasured variables — present as a generalizability adjustment with explicit assumptions, never as a true probability-design weight.

Worked HEOR example (MEPS national per-capita cost with correct domain estimation)

Question: mean annual total healthcare expenditure among U.S. adults with diagnosed diabetes, for a budget-impact model denominator. MEPS is a stratified, multistage probability sample; the person-level file carries `PERWT` (person weight), `VARSTR` (variance stratum), and `VARPSU` (variance PSU). (1) Build the design object on the full file using `PERWT`, `VARSTR`, `VARPSU` with nested-PSU handling and single-PSU-stratum adjustment. (2) Define the diabetes domain with an indicator (e.g., from the condition file mapped to the person via `DUPERSID`), keeping every row. (3) Estimate the domain mean of total expenditure (`TOTEXP`) via `svymean`/`subset(design,...)` (R) or `PROC SURVEYMEANS ... DOMAIN diabetes` (SAS), which yields the population mean and a Taylor-linearized SE that respects strata and clusters. The wrong way — `df[df.diabetes==1]` then a weighted mean — returns the same point estimate but a materially smaller, invalid SE because it discards the design information for excluded PSUs; in MEPS-scale data this routinely understates the SE by 20–40%, narrowing a 95% CI enough to misstate affordability. Multiply the design-based per-capita mean by the population denominator for the budget-impact input, and propagate the design-based SE into the model's probabilistic sensitivity analysis rather than an i.i.d. SE.

Interpreting the output

Consider a 10-respondent NHANES-like sample: 4 non-Hispanic Black respondents each with a sampling weight of 500, 4 Hispanic respondents each with a weight of 500, and 2 non-Hispanic White respondents each with a weight of 2,000. Unweighted mean SBP = 126.0 mmHg. Weighted mean SBP = 123.75 mmHg (weighted sum divided by total weight of 8,000).

(1) Formal statistical interpretation. The weighted estimate of 123.75 mmHg is a design-consistent estimator of the population mean: each observation is scaled to represent the number of people in the target population it stands for. The unweighted estimate of 126.0 mmHg is biased because the two high-weight non-Hispanic White respondents are under-represented in the sample relative to the population. Uncertainty for the weighted mean must be quantified with Taylor linearization or balanced repeated replication to correctly account for the design effect; a simple i.i.d. standard error ignores clustering and stratification and understates the true sampling variability.

(2) Practical interpretation for a decision-maker. The 2.25 mmHg difference between the weighted (123.75) and unweighted (126.0) estimates reflects the actual racial and ethnic composition of the target population, not the sample composition. Using the unweighted mean for a national prevalence estimate or a budget-impact model would misrepresent the population burden. When the goal is a population-level statement — rather than a description of who happened to be surveyed — always apply survey weights and report the design-based standard error, not the simple standard error.

Worked example

Scenario

A national health survey measures systolic blood pressure (SBP) in U.S. adults. To get reliable estimates for smaller racial/ethnic groups, the survey oversamples Non-Hispanic Black and Hispanic adults — meaning more of them are recruited relative to their true share of the population. Non-Hispanic White adults, the largest group, need far fewer additional recruits to be measured reliably. After the data are collected, the survey assigns each respondent a sampling weight. Here we have 10 respondents: 4 Non-Hispanic Black (each representing 500 people in the country), 4 Hispanic (each representing 500), and 2 Non-Hispanic White (each representing 2,000). We want the national mean SBP — not the mean of the recruited sample.

Dataset

Ten respondents from a complex survey, each assigned a sampling weight based on their group's selection probability.

Steps

• Unweighted mean (wrong for a national estimate): add all 10 SBP readings and divide by 10. Sum = 130+132+128+130+126+124+125+125+120+120 = 1,260. Unweighted mean = 1,260 / 10 = 126.0 mmHg. This treats every respondent as equally representative, but the two NH White respondents each stand for 2,000 people while the eight minority respondents each stand for only 500 — so the unweighted mean over-represents the minority groups.

• Weighted mean (correct national estimate): multiply each respondent's SBP by their sampling weight, sum those products, then divide by the total weight. Weighted sum = (130+132+128+130) x 500 + (126+124+125+125) x 500 + (120+120) x 2,000 = 520 x 500 + 500 x 500 + 240 x 2,000 = 260,000 + 250,000 + 480,000 = 990,000.

• Total weight = 4 x 500 + 4 x 500 + 2 x 2,000 = 2,000 + 2,000 + 4,000 = 8,000. This is the total number of population members represented by these 10 respondents.

• Weighted mean SBP = 990,000 / 8,000 = 123.75 mmHg. The two NH White respondents each represent 2,000 people (combined 4,000 of the 8,000 total weight), so the lower NH White SBP pulls the national estimate down relative to the unweighted value.

Result

Unweighted mean = 126.0 mmHg; weighted mean = 990,000 / 8,000 = 123.75 mmHg. The 2.25 mmHg gap arises entirely from oversampling: minority groups with higher average SBP make up 8 of 10 respondents but only 4,000 of 8,000 population units. The weighted estimate is the correct national figure; the unweighted one would overstate mean population SBP.

Runnable example

import pandas as pd
from samplics.estimation import TaylorEstimator

# df already loaded, full file (do NOT pre-filter to the diabetes domain).
# Overall national mean expenditure (population estimand, design-based variance):
overall = TaylorEstimator(param="mean")
overall.estimate(
    y=df["totexp"],
    samp_weight=df["perwt"],
    stratum=df["varstr"],
    psu=df["varpsu"],
)
print("National mean:", overall.point_est, "SE:", overall.stderror)

# Domain (diabetes) estimate: pass the domain to keep all PSUs/strata in the variance.
# This is the CORRECT subgroup approach; filtering df first would drop PSUs and break the SE.
domain = TaylorEstimator(param="mean")
domain.estimate(
    y=df["totexp"],
    samp_weight=df["perwt"],
    stratum=df["varstr"],
    psu=df["varpsu"],
    domain=df["diabetes"],
)
print(domain.point_est)   # keyed by domain level (0/1)
print(domain.stderror)    # design-based SE for each domain, valid for the budget-impact denominator

library(survey)
options(survey.lonely.psu = "adjust")  # robust handling of single-PSU strata

des <- svydesign(
  ids     = ~varpsu,      # PSU / cluster
  strata  = ~varstr,      # variance stratum
  weights = ~perwt,       # person-level sampling weight
  nest    = TRUE,         # PSUs nested within strata (MEPS/NHANES convention)
  data    = meps
)

# National per-capita expenditure (population mean + design-based SE):
svymean(~totexp, des)

# CORRECT domain estimation: subset the DESIGN, not the data frame, so excluded
# PSUs still contribute to the between-PSU variance.
dm <- subset(des, diabetes == 1)
svymean(~totexp, dm)               # population mean + valid Taylor-linearized SE
svyglm(totexp ~ age + sex, design = dm)   # population-representative regression

# Survival on weighted survey data (e.g., time to a costly event):
# svycoxph(Surv(time, event) ~ arm, design = des)