Inferential Statistics Foundations

What statistical inference is, and what it is not

Statistical inference is the formal process of using data from a sample to reason about an unobserved population. A sample is whatever we can actually see; the population is every person, fill, or episode we would like to say something about. Because we cannot observe everyone, every estimate we produce carries sampling variability — the fact that a different sample from the same population would give a slightly different answer. Inferential methods quantify that variability so that consumers of evidence can judge how much weight to place on a number. Crucially, this machinery assumes the only source of error is sampling; in observational data, bias — confounding, selection, measurement error — usually dominates, a point so important for RWE that it is returned to throughout this entry.

Point estimates and the standard error vs standard deviation distinction

A point estimate is the single best-guess value for a quantity of interest computed from the data: a mean, a proportion, a risk difference, a hazard ratio. It summarises the sample in one number, but says nothing by itself about uncertainty. Uncertainty is quantified by the standard error (SE), which is the standard deviation of the point estimate's sampling distribution — in other words, how spread out the estimate would be if we repeated the study many times. The SE is not the same as the standard deviation (SD) of the raw data, and confusing the two is one of the most common errors in applied work.

The standard deviation describes variability between individual observations in the sample — it answers "how different are patients from one another?" The standard error describes variability of the summary statistic — it answers "how different would our mean be if we repeated the study?" For a sample mean the relationship is SE = SD / √n, which makes explicit that the SE shrinks as n grows while the SD does not: with more data the estimate becomes more precise even though patients remain just as heterogeneous as before. State this distinction twice because it matters every time a CI or test statistic is built: the SE is what goes in the denominator, not the SD.

Confidence intervals — correct interpretation and the practical RWE reading

A 95% confidence interval (CI) is a range constructed by a procedure that, over many repeated samples, would contain the true parameter value in 95% of cases. This is the frequentist long-run coverage interpretation. The correct reading: "the data are compatible with any value inside this interval under the assumptions of the model." The common misreading: "there is a 95% probability the true value is inside this interval" — that statement assigns probability to the parameter, which in frequentist statistics is a fixed (if unknown) constant, not a random variable.

For RWE reviewers, the practical reading is: the CI is a range of effect sizes compatible with the data; any value inside it cannot be ruled out on the evidence alone. A wide CI says the study is uninformative; a narrow CI that still excludes the null is informative and supports a conclusion. The width of the CI is driven by two factors: (1) sample size — larger samples shrink the SE and thereby narrow the interval; and (2) the variance in the outcome — noisier data widen the interval regardless of n. In claims databases, n can reach millions, so CIs can become narrow enough to declare statistical significance for effects so small they have no clinical meaning whatsoever.

Hypothesis testing machinery

The Neyman–Pearson framework formalises a decision: either reject a null hypothesis H₀ or fail to reject it. The null is typically "no effect" (difference = 0, ratio = 1). The alternative hypothesis H₁ specifies the direction or range of effects considered meaningful. The analyst computes a test statistic — a number that measures how far the observed data are from what the null predicts, scaled by the SE: z = (estimate − null value) / SE. Large absolute values of z are unlikely under H₀.

The p-value is the probability of observing a test statistic at least as extreme as the one computed, if the null hypothesis were true and all model assumptions held. A small p-value means the data would be surprising if the null were true; it does not mean:

• the null is false (it is not the probability that H₀ is true),
• the effect is large or clinically important,
• the result will replicate,
• the analysis is free of bias, or
• any particular model assumption is satisfied.

The threshold α = 0.05 is a convention borrowed from mid-twentieth-century experimental science, not a law of nature. A p-value of 0.049 and a p-value of 0.051 are not meaningfully different; treating them as a hard gate between "significant" and "not significant" is called dichotomania and is a primary source of irreproducible findings.

Type I and Type II errors, power, and multiplicity

A Type I error (false positive, α) occurs when we reject a null hypothesis that is actually true: we declare an effect when there is none. The conventional α = 0.05 means we accept a 5% chance of this error in a single test. A Type II error (false negative, β) occurs when we fail to reject a null that is false: we miss a real effect. Statistical power (1 − β) is the probability of correctly detecting a real effect of a specified size; 80% or 90% power is the conventional target, meaning we accept a 10–20% chance of missing the effect.

Multiplicity

is the inflation of the Type I error rate when many hypotheses are tested simultaneously. With 20 independent tests at α = 0.05, we expect one false positive by chance alone. In pharmacoepidemiological safety surveillance, where dozens of outcomes are screened across many drugs, multiplicity corrections (Bonferroni, Benjamini–Hochberg false discovery rate) or sequential probability ratio methods (maxSPRT) are essential. Pre-specification of the primary hypothesis and secondary hypotheses is the cleanest guard.

Statistical significance is not clinical relevance — the large-database RWE trap

This distinction is perhaps the most important applied lesson in RWE statistics. In a database with n = 2,000,000 patients, a true mean difference of 0.01 units (clinically negligible) will produce a p-value far below 0.001 and a 95% CI that excludes zero by a comfortable margin. The result is "statistically significant" in the strict sense — unlikely under the null — but completely unimportant for clinical or policy decisions. Conversely, a study that is underpowered may miss a clinically meaningful effect. Effect size plus CI width is always what matters; the p-value is a binary signal about the null, not a scale for importance.

The ASA's 2016 statement (Wasserstein & Lazar) and its 2019 follow-up formalise this principle: do not use p < 0.05 as the sole arbiter of evidence; report and interpret the effect size and its CI; and consider the entire distribution of compatible effects, not just whether the null is excluded. The estimation-first culture now favoured by major journals and regulatory bodies (FDA, EMA) pre-specifies estimands and target effect measures, then reports them with intervals, rather than framing the study as a significance test.

Pros, cons, and trade-offs

• Confidence intervals vs p-values only. CIs give the same information as the p-value (zero inside = significant at the matching α) but add the effect size and the range of compatible values, which a bare p-value hides. There is no disadvantage to reporting CIs; there is a large disadvantage to reporting only the p-value. Prefer CIs as the primary reporting form for all RWE; always accompany with the point estimate and its units.
• Two-sided vs one-sided tests. A two-sided test asks whether the effect differs from the null in either direction (α = 0.05 split as 2.5% per tail); a one-sided test specifies the direction in advance and uses the full 5% in one tail, giving a lower critical z (1.645 vs 1.96). One-sided tests are only justified when a result in the opposite direction would have no consequences; regulatory submissions almost always require two-sided tests.
• Frequentist vs Bayesian inference. Frequentist inference (CIs, p-values) reports what the data are compatible with under repeated sampling; it does not produce a probability statement about the parameter. Bayesian inference produces a posterior probability distribution over the parameter given the data and a prior, and is natural for decision-analytic settings (HTA, value of information). Frequentist methods dominate regulatory RWE submissions; Bayesian approaches are growing in HTA and adaptive trial settings. The foundations described here are the frequentist primitives that underlie nearly all RWE analysis pipelines.
• Unadjusted vs model-based inference. A two-sample t-test is valid only if the groups are exchangeable (no confounding). In observational data they almost never are, so the raw inferential machinery must be embedded in a regression or weighting framework that accounts for confounders before the SE and CI are meaningful. A p-value from an unadjusted comparison in an unbalanced observational cohort is not a valid test of the exposure–outcome null.

When NOT to use — and when it is actively misleading or dangerous

• p-values as variable selection or balance checks. Selecting confounders to include in a model because they have p < 0.05 in a univariate screen (or because a balance test after propensity weighting is "significant") is a methodological error: it conflates statistical significance with confounding importance, biases the resulting estimates, and has no theoretical justification. Use subject-matter knowledge and standardised mean differences for balance assessment, not hypothesis tests.
• Significance chasing and dichotomania. Stopping an analysis when p < 0.05 and reporting only those outcomes with significant results — or adjusting analytical choices until significance is achieved — is p-hacking, which inflates the false-positive rate beyond α regardless of what the individual test threshold is.
• Post-hoc power calculations. Computing power from the observed effect size after a null result has been obtained is circular: observed power is a deterministic function of the observed p-value and tells the reader nothing beyond what the p-value already communicated. After the study is complete, report the observed effect size, its CI, and the smallest effect the study could have detected at the target power — not the "observed power."
• Treating statistical inference as a substitute for bias analysis. A tight CI around a biased estimate is precisely wrong — more data makes the wrong answer more certain. In observational databases, confounding, selection bias, and measurement error often dominate sampling variability, especially for large n. A p-value below any threshold cannot rescue a design that fails to control for confounding; and a significant result in a large administrative database says only that sampling error can be ruled out, not that the effect is causal. This is the bridge to the catalog's confounding-control and bias-analysis entries.

Data-source operational depth

• Claims (FFS commercial / Medicare FFS): With millions of person-rows, nearly any nonzero effect will be "significant" — the practical question in claims analysis is always effect size and CI width, not p-value. The effective sample size for adjusted analyses is far smaller than the nominal n because of continuous-enrollment requirements, washout erosion, and IPTW design effects (effective n ≈ nominal n / (1 + CV²(weights))). Report the effective n alongside the point estimate and CI. In Medicare FFS, differential administrative censoring by exposure arm — from higher death rates in frailer patients or from Medicare Advantage leakage — means the standard survival-analysis SE assumptions may be violated; the SE and CI must account for this.
• EHR: Visit-driven capture creates informative missingness that the standard complete-case SE does not account for. Multiple imputation or inverse-probability-of-observation weighting is required before the SE is valid. Clustering within sites (multi-site EHR networks) inflates the SE; a naive pooled-dataset SE that ignores site clustering understates uncertainty.
• Registry: Typically smaller and more selective than claims; the precision framing (CI half-width) is usually the binding constraint rather than hypothesis-test power. Non-random enrollment in registries means the SE addresses only sampling variability within the enrolled group, not representativeness to the broader population — a narrow CI in a highly selected registry may not transport.
• Linked claims–EHR–vital records: Linkage selection (only the linkable subset is analysed) changes the population to which the SE and CI apply. The SE is correct for the linked subset; transportability to the unlinked remainder requires additional assumptions.

Interpreting the output

Consider a two-arm study comparing antihypertensive regimens in a claims-based cohort: 10 patients per arm, observed mean SBP reductions of 12.0 mmHg (Arm A) and 8.0 mmHg (Arm B). The analysis returns: mean difference = 4.0 mmHg, SE = 1.0, 95% CI [2.04, 5.96], z = 4.0, p < 0.001.

(1) Formal statistical interpretation. The point estimate of 4.0 mmHg is the observed difference in mean SBP reduction. The 95% CI [2.04, 5.96] is produced by a procedure that, if the study were repeated under identical conditions many times, would contain the true mean difference in approximately 95% of those replications; values of the true difference between 2.04 and 5.96 mmHg are compatible with the observed data at the 5% significance level. The p-value < 0.001 is the probability — under the null hypothesis that the true difference is exactly zero — of observing a difference at least as large as 4.0 mmHg in absolute value; it is not the probability that the null hypothesis is true.

(2) Practical interpretation for a decision-maker. Arm A reduced systolic blood pressure by roughly 4 more millimeters of mercury than Arm B, and the entire plausible range (2.0–6.0 mmHg) falls on the side favoring Arm A. Whether a 4 mmHg difference crosses the threshold of clinical importance depends on each patient's baseline risk, comorbidities, and tolerability — statistical significance alone does not establish that the difference is large enough to change treatment decisions.

Decision diagram

flowchart TD
  S[Sample data<br/>n observations] --> PE[Point estimate<br/>e.g. mean difference = 4.0 mmHg]
  S --> SE[Standard error<br/>SE = sqrt&#40;s²/n_A + s²/n_B&#41; = 1.0]
  PE --> CI[95% Confidence interval<br/>estimate ± 1.96 × SE = 2.04 to 5.96]
  PE --> TS[Test statistic<br/>z = estimate / SE = 4.0]
  TS --> PV[p-value<br/>P&#40;|z| ≥ 4.0 | H₀ true&#41; < 0.001]
  CI --> INT[Interpretation<br/>range of effects compatible with data<br/>zero is not inside → reject H₀]
  PV --> INT
  INT --> ES[Effect size + CI width<br/>is what matters for RWE decisions<br/>not p-value alone]

flowchart LR
  T[True state of the world]
  T -->|H₀ is TRUE| A[Reject H₀<br/>Type I error α<br/>false positive]
  T -->|H₀ is TRUE| B[Fail to reject H₀<br/>Correct decision<br/>1 - α]
  T -->|H₀ is FALSE| C[Reject H₀<br/>Correct decision<br/>Power = 1 - β]
  T -->|H₀ is FALSE| D[Fail to reject H₀<br/>Type II error β<br/>false negative]

Worked example

Scenario

A hospital quality team compares systolic blood pressure (SBP, in mmHg) reductions over 12 weeks for two antihypertensive drugs — Drug A (n = 10 patients) and Drug B (n = 10 patients). Drug A produces a mean reduction of 12 mmHg; Drug B produces a mean reduction of 8 mmHg. In both groups the variance of individual reductions is 5 mmHg². The team wants the point estimate of the difference, the standard error, a 95% confidence interval, and a test statistic to decide whether the difference is statistically significant.

Dataset

Summary statistics for a two-group blood-pressure reduction study (10 patients per arm).

FIG. 1 — DESIGN TIMELINE

Steps

Result

Point estimate = 4.0 mmHg; SE = 1.0 mmHg; 95% CI = [2.04, 5.96] mmHg; z = 4.0; p < 0.001. The interval excludes zero, so the result is statistically significant at alpha = 0.05. The effect size (4 mmHg) and CI width (about 4 mmHg) together characterise the finding — the p-value alone does not.

Trade-offs

Runnable example

import numpy as np
from scipy import stats

# --- From summary statistics (known n, mean, variance) ---
n_a, n_b = 10, 10
mean_a, mean_b = 12.0, 8.0
var_a, var_b = 5.0, 5.0     # variance of individual observations (SD^2), NOT SE^2

diff = mean_a - mean_b                          # point estimate of mean difference
se   = np.sqrt(var_a / n_a + var_b / n_b)       # SE of the difference: sqrt(s²/n + s²/n)
# Note: se != sqrt(var_a) or sqrt(var_b) — SD of the raw data is sqrt(var), SE is SD/sqrt(n)

z    = diff / se                                # z-statistic (large n; use t for small n)
ci_lo = diff - 1.96 * se                        # 95% CI lower bound
ci_hi = diff + 1.96 * se                        # 95% CI upper bound
p_val = 2 * (1 - stats.norm.cdf(abs(z)))        # two-sided p-value

print(f"Point estimate : {diff:.2f} mmHg")
print(f"SD of raw data : {np.sqrt(var_a):.3f} mmHg  (variability between individual patients)")
print(f"SE of estimate : {se:.3f} mmHg  (variability of the mean difference across repeated studies)")
print(f"95% CI         : [{ci_lo:.2f}, {ci_hi:.2f}] mmHg")
print(f"z-statistic    : {z:.2f}")
print(f"p-value        : {p_val:.4f}")

# --- From individual-level data (Welch's t-test, recommended when variances may differ) ---
rng = np.random.default_rng(42)
group_a = rng.normal(loc=mean_a, scale=np.sqrt(var_a), size=n_a)
group_b = rng.normal(loc=mean_b, scale=np.sqrt(var_b), size=n_b)
t_stat, p_ttest = stats.ttest_ind(group_a, group_b, equal_var=False)  # Welch's t-test
ci_ttest = stats.t.interval(0.95, df=len(group_a)+len(group_b)-2,
                            loc=group_a.mean()-group_b.mean(),
                            scale=stats.sem(group_a-group_b))
print(f"\nWelch t-test on simulated data: t={t_stat:.2f}, p={p_ttest:.4f}")
print(f"t-test 95% CI: [{ci_ttest[0]:.2f}, {ci_ttest[1]:.2f}]")

# --- From summary statistics ---
n_a <- 10L; n_b <- 10L
mean_a <- 12.0; mean_b <- 8.0
var_a  <- 5.0;  var_b  <- 5.0   # variance of raw observations (SD^2), NOT the SE

diff   <- mean_a - mean_b                      # point estimate
se_raw <- sqrt(var_a / n_a + var_b / n_b)      # SE of mean difference: sqrt(s^2/n + s^2/n)
# SD of raw data vs SE of estimate:
cat("SD of individual observations:", sqrt(var_a), "mmHg\n")
cat("SE of mean-difference estimate:", se_raw, "mmHg  (shrinks with n; SD does not)\n")

z      <- diff / se_raw
ci_lo  <- diff - 1.96 * se_raw
ci_hi  <- diff + 1.96 * se_raw
p_z    <- 2 * pnorm(-abs(z))

cat(sprintf("Point estimate : %.2f mmHg\n", diff))
cat(sprintf("SE             : %.3f mmHg\n", se_raw))
cat(sprintf("95%% CI         : [%.2f, %.2f] mmHg\n", ci_lo, ci_hi))
cat(sprintf("z-statistic    : %.2f\n", z))
cat(sprintf("p-value (z)    : %.4f\n\n", p_z))

# --- From individual-level data using base R t.test ---
set.seed(42)
group_a <- rnorm(n_a, mean = mean_a, sd = sqrt(var_a))
group_b <- rnorm(n_b, mean = mean_b, sd = sqrt(var_b))
tt      <- t.test(group_a, group_b, var.equal = FALSE)   # Welch's t-test
cat("t.test result:\n"); print(tt)

# --- Large-database demonstration: the 'significant but tiny' RWE trap ---
n_large  <- 500000L                     # half a million patients per arm (realistic claims n)
diff_tiny <- 0.05                       # 0.05 mmHg true difference — clinically negligible
se_large  <- sqrt(var_a / n_large + var_b / n_large)
z_large   <- diff_tiny / se_large
p_large   <- 2 * pnorm(-abs(z_large))
ci_large  <- c(diff_tiny - 1.96 * se_large, diff_tiny + 1.96 * se_large)
cat(sprintf("\n[Large-n demo] Diff = %.3f mmHg, SE = %.5f, z = %.1f, p = %.2e\n",
            diff_tiny, se_large, z_large, p_large))
cat(sprintf("95%% CI = [%.4f, %.4f] mmHg  -> significant but trivially small\n",
            ci_large[1], ci_large[2]))
cat("Lesson: in claims databases p < 0.05 is nearly automatic; report effect size + CI.\n")

/* --- From summary statistics (DATA step) --- */
data summary_calc;
  n_a = 10; n_b = 10;
  mean_a = 12.0; mean_b = 8.0;
  var_a  = 5.0;  var_b  = 5.0;   /* variance of raw observations, NOT the SE */

  diff   = mean_a - mean_b;                   /* point estimate = 4.0 */
  sd_raw = sqrt(var_a);                        /* SD of raw data: 2.236 mmHg */
  se     = sqrt(var_a/n_a + var_b/n_b);        /* SE of estimate: sqrt(0.5+0.5)=1.0 mmHg */
  z      = diff / se;                           /* z = 4.0 / 1.0 = 4.0 */
  ci_lo  = diff - 1.96 * se;                   /* 95% CI lower = 4.0 - 1.96 = 2.04 */
  ci_hi  = diff + 1.96 * se;                   /* 95% CI upper = 4.0 + 1.96 = 5.96 */
  p_val  = 2 * (1 - probnorm(abs(z)));          /* two-sided p-value */

  put "Point estimate = " diff " mmHg";
  put "SD (raw data)  = " sd_raw " mmHg  [between-patient variability; does NOT shrink with n]";
  put "SE (estimate)  = " se    " mmHg  [variability of the mean diff; shrinks with n]";
  put "95% CI         = [" ci_lo ", " ci_hi "] mmHg";
  put "z-statistic    = " z;
  put "p-value        = " p_val;
run;

/* --- From individual-level data using PROC TTEST (Satterthwaite / Welch default) --- */
/* Simulate individual-level data for the example */
data indiv;
  call streaminit(42);
  do i = 1 to 10; sbp_reduction = rand('Normal', 12.0, sqrt(5.0)); group = 'A'; output; end;
  do i = 1 to 10; sbp_reduction = rand('Normal',  8.0, sqrt(5.0)); group = 'B'; output; end;
run;

proc ttest data=indiv;
  class group;
  var sbp_reduction;
  /* Default: Satterthwaite (Welch) t-test; use pooled option for equal-variance assumption */
run;

/* --- Large-n demonstration: claims-database 'significant but tiny' trap --- */
%macro large_n_demo(n=500000, diff_tiny=0.05, var=5.0);
  data _null_;
    n         = &n;
    diff      = &diff_tiny;
    se_large  = sqrt(2 * &var / n);   /* SE with equal n and equal variance */
    z_large   = diff / se_large;
    p_large   = 2 * (1 - probnorm(abs(z_large)));
    ci_lo     = diff - 1.96 * se_large;
    ci_hi     = diff + 1.96 * se_large;
    put "=== Claims-database large-n demo (n=" n ") ===";
    put "True diff = " diff " mmHg (clinically negligible)";
    put "SE        = " se_large;
    put "z         = " z_large;
    put "p-value   = " p_large;
    put "95% CI    = [" ci_lo "," ci_hi "] -> p<0.05 but effect is trivially small";
    put "Lesson: report effect size + CI width, not just p-value.";
  run;
%mend large_n_demo;
%large_n_demo;

Citations

1. [1]Wasserstein RL, Lazar NA. The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician. 2016;70(2):129-133.
2. [2]Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology. 2016;31(4):337-350.

1. [3]Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature. 2019;567(7748):305-307.

1. [4]Wasserstein RL, Schirm AL, Lazar NA. Moving to a World Beyond 'p < 0.05'. The American Statistician. 2019;73(sup1):1-19.