Chi-Square Test of Independence

In plain language

The chi-square test of independence checks whether two categorical variables — such as treatment group and whether a patient was hospitalized — are related or just coincidentally differ between groups. It works by comparing what you actually counted in each cell of a table (for example, 8 readmissions in the treated group and 18 in the control group) to what you would expect to see if there were no association at all, then summarising those discrepancies in a single number. A large discrepancy produces a small p-value, which means the pattern is unlikely to be due to chance — but the test cannot tell you how large or important the difference is, so you always need to also report a concrete effect size such as the risk difference ("10 fewer events per 100 patients") or the relative risk.

What the chi-square test of independence actually does

Given a contingency table with r rows (groups) and c columns (outcome categories), the test compares the observed count in every cell to the count that would be expected if the row variable and the column variable were statistically independent. The expected count for cell (i, j) is:

E(i,j) = (row i total × column j total) / grand total

The Pearson chi-square statistic sums the squared, scaled discrepancies across all cells:

χ² = Σ [ (O - E)² / E ]

Under the null hypothesis of independence this statistic follows (approximately) a chi-squared distribution with degrees of freedom (r − 1)(c − 1). A large χ² signals that at least one cell count is further from its expected value than pure chance would produce. The p-value is the probability of observing a χ² at least this large if independence were truly the case.

The chi-squared approximation is valid when the expected counts are large enough for the Central Limit Theorem to have kicked in. The widely cited rule of thumb — every expected cell count should be ≥ 5 — is a practical heuristic, not a theorem. Campbell (2007) recommends the more precise criterion that no more than 20 % of expected counts fall below 5 and none falls below 1. When this criterion is violated, Fisher's exact test is preferred for 2 × 2 tables; for larger tables, the options include collapsing sparse categories, a mid-p correction, or exact methods in software.

Degrees of freedom and table structure

For a 2 × 2 table (two groups, binary outcome), df = 1. For a 3 × 2 table (three groups, binary outcome), df = 2. For a 4 × 3 table, df = 6. The degrees of freedom capture how many cells are free to vary once the row and column totals are fixed. Increasing df reduces the chance of a large χ² under the null, so a given χ² value corresponds to a larger p-value as the table grows.

Yates' continuity correction — the debate

For 2 × 2 tables specifically, Yates (1934) proposed subtracting 0.5 from each |O − E| before squaring, bringing the discrete chi-square distribution closer to the continuous chi-squared curve. The Yates correction makes the test more conservative (larger p-values). The modern consensus, reviewed by Serra and Rea (2022), is that the correction is over-conservative: it produces p-values larger than the true exact p-value from Fisher's exact test, and Fisher's exact test itself is the preferred alternative when expected counts are small. Most analysts therefore do not apply Yates' correction as a default; when expected counts are small, they switch to Fisher's exact test entirely. In software, the `correction=False` flag in `scipy.stats.chi2_contingency` and `correct=FALSE` in `chisq.test` disable Yates' correction and are the appropriate defaults when expected counts are adequate.

Chi-square for trend — Cochran-Armitage

When the column categories are ordered (e.g., dose groups: none, low, medium, high), the standard chi-square test discards that ordering and tests only whether any pattern of association exists. The Cochran-Armitage trend test uses the ordering explicitly to construct a more powerful one-degree-of-freedom test for a monotone dose-response relationship. In pharmacoepidemiology and benefit-risk assessment, where ordered exposure categories are common, the trend test is often more appropriate than a general chi-square; it also produces a cleaner narrative — "risk increases with dose" — rather than the omnibus "at least one group differs."

Effect sizes: the deliverable that chi-square alone cannot provide

The chi-square statistic and its p-value answer only one question: is there evidence against independence? They do not tell you whether the association is clinically or economically meaningful. For binary outcomes in a 2 × 2 table, the effect size choices are:

• Risk difference (RD): proportion with event in group A minus proportion in group B.
• Relative risk (RR): event rate in group A divided by event rate in group B.
• Odds ratio (OR): the odds of the event in A divided by odds in B. Approximates the RR
• Phi coefficient / Cramér's V: standardized chi-square-based effect size. Phi applies to

For HEOR and HTA submissions, risk difference and relative risk (with 95 % confidence intervals) are the effect measures that connect most directly to budget-impact models, absolute risk reduction, and number-needed-to-treat calculations. A chi-square p-value without these effect estimates is insufficient for a decision-making audience.

RWE realities: the large-n problem and why p-values mislead at claims scale

In real-world evidence studies using administrative claims or EHR data, sample sizes of 50,000 to several million patients are routine. At this scale, the chi-square test will reject the null for association differences that are substantively meaningless. A 0.2 percentage-point difference in comorbidity prevalence between two cohorts of 500,000 patients will yield χ² values in the hundreds and p-values indistinguishable from zero. This is not an error in the test — it is working exactly as designed. The test is highly powered to detect any departure from independence, no matter how trivial.

The practical implication is that in large observational datasets, p-values from chi-square tests should be treated as nuisance outputs, not deliverables. What matters is:

1. The absolute risk difference and its 95 % confidence interval, sized against a pre-specified minimally important difference. 2. The standardized difference (also called the standardized mean difference for binary variables, where it equals the risk difference divided by the pooled standard deviation of the proportion). A standardized difference < 0.10 indicates negligible imbalance for covariate balance assessment, regardless of n.

For Table 1 balance comparisons in observational RWE studies, standardized differences are now the preferred summary because, unlike chi-square p-values, they do not inflate with sample size. Reporting chi-square p-values in a Table 1 comparing a treated group of 200,000 to a control group of 200,000 will produce a list of "statistically significant" differences in 80 %+ of covariates even after careful propensity-score matching — this is uninformative and potentially misleading.

Independence assumption violations in real-world data

The chi-square test assumes that each observation (each row in the contingency table) is independent. This assumption is violated in several common RWE settings:

• Claim-level rather than patient-level analysis: a patient with 12 hospitalizations
• Clustering within providers or facilities: patients treated by the same physician or at
• Repeated binary outcomes per patient over time: if a patient can have the outcome in

Pros, cons, and trade-offs

Pros: - Conceptually simple: observed vs expected, with a single formula most analysts can verify by hand for a 2 × 2 table. - Valid under minimal assumptions when expected counts are adequate (≥ 5 in each cell). - Available in every statistical package; the implementation is identical across software. - Extends naturally to r × c tables with a single statistic and a single p-value. - At large n, the approximation is excellent; the test is computationally trivial. - The Cochran-Armitage extension handles ordered categorical predictors without additional modelling machinery.

Cons: - Does not produce a clinically meaningful effect estimate; risk difference, RR, or OR must be computed separately and are the actual deliverable for decision-makers. - The chi-squared approximation breaks down when expected counts are small — the p-value is unreliable in sparse tables. - Completely powered at large n: trivial associations are flagged as statistically significant; p-values alone are uninterpretable at claims scale. - Cannot accommodate covariates or confounding adjustment; it is a marginal (unadjusted) test. For adjusted inference, route to logistic regression or log-binomial regression. - Does not distinguish the direction of the association — a large χ² could reflect excess events in group A or in group B; the contingency table must be inspected. - Sensitive to the unit of analysis: claim-level rows rather than patient-level rows produce an anti-conservative test.

Trade-offs vs alternatives: - Fisher's exact test: exact p-value from the hypergeometric distribution; no large-sample approximation needed; preferred when expected counts are small; computationally heavier for large tables (though modern software handles this for typical RWE table sizes). - McNemar's test: the paired-data analogue for a 2 × 2 table where the two rows represent the same subjects measured twice (e.g., pre/post or matched pairs). Using a standard chi-square test on paired binary data is a common and consequential mistake. - Logistic regression: extends chi-square from a bivariate marginal test to a multivariable adjusted model; produces an OR with CI; handles continuous covariates; is the adjusted version of the chi-square test and the preferred primary method for binary outcomes in confounded observational comparisons.

When to use

Use the chi-square test of independence when:

• The outcome is binary or multinomial (categorical with no intrinsic ordering or an
• The unit of analysis is the patient (or other independent observational unit), not the
• All expected cell counts satisfy the ≥ 5 rule (or more precisely, the Campbell criterion).
• The goal is a simple bivariate test of association — e.g., comparing 30-day readmission
• A baseline Table 1 comparison in an RCT (where balance testing is legitimately of interest
• As a sensitivity check alongside an adjusted logistic or log-binomial regression model for
• For ordinal categorical predictors where you want to test a monotone trend, use the

When NOT to use — and when it is actively misleading

• Paired data (pre-post or matched pairs): if the two rows of the table represent the same
• Small expected counts: when any expected cell count falls below 5 (or more than 20 % of
• Claim-level rather than patient-level units: building the contingency table from claim
• As evidence of a causal association: chi-square tests for statistical association, not
• As the primary analysis for confounded comparisons: in any observational design without
• At large n for substantive decision-making: at claims scale (hundreds of thousands of
• Ordered categories without exploiting the order: if the columns represent an ordered

Interpreting the output

In the worked example, a discharge coordination program is compared with usual care on 30-day readmission in a 2×2 contingency table. The Pearson chi-square statistic is approximately 9.52 on df = 1, with p ≈ 0.002. The estimated risk difference is −0.20 (20 fewer readmissions per 100 patients) and the relative risk is 0.50 (half the readmission rate in the program arm relative to usual care).

(1) Formal interpretation. The chi-square statistic of approximately 9.52 is the weighted sum of squared discrepancies between observed and expected cell counts. Under the null hypothesis of no association between program assignment and readmission, a statistic this large or larger on df = 1 arises by chance in approximately 0.2% of samples (p ≈ 0.002). The test confirms that the observed association is incompatible with independence at conventional significance levels. The p-value alone does not quantify the magnitude of the association — the risk difference (−0.20, i.e., 20 fewer readmissions per 100 patients) and the relative risk (0.50) are the effect measures that carry clinical and economic meaning.

(2) Practical interpretation. The data are consistent with the discharge program being associated with roughly half the 30-day readmission rate of usual care — a risk difference of 20 per 100 that is substantial in most hospital settings. The chi-square p-value confirms the association is unlikely to be pure chance; it does not establish causation. Because this is an unadjusted comparison, differences in baseline risk, patient mix, and facility factors between the two groups may confound the estimate. An adjusted logistic regression model using the same outcome is the appropriate next step for confirmatory inference.

Worked example

Scenario

A health outcomes analyst is comparing 30-day hospital readmission rates between patients who received a new discharge coordination program (Program group) and patients who received usual care (Control group). Each group has 100 patients. The analyst builds a 2x2 contingency table, computes expected counts, calculates the chi-square statistic by hand, and reports the risk difference and relative risk alongside the p-value.

Dataset

30-day readmission counts for 100 patients per group. Totals chosen so every expected count is a whole number and each (O-E)^2/E term is an exact fraction.

Steps

• Compute expected counts using E = (row total x column total) / grand total. E(Program, readmitted) = (100 x 60) / 200 = 6000 / 200 = 30. E(Program, not readmitted) = (100 x 140) / 200 = 14000 / 200 = 70. E(Control, readmitted) = (100 x 60) / 200 = 6000 / 200 = 30. E(Control, not readmitted) = (100 x 140) / 200 = 14000 / 200 = 70.

• All four expected counts equal 30 or 70, both well above 5. The chi-square approximation is valid.

• Compute the squared deviation divided by expected for each cell. Cell (Program, readmitted): difference = 20 - 30 = -10; squared = 100; term = 100/30. Cell (Program, not readmitted): difference = 80 - 70 = 10; squared = 100; term = 100/70. Cell (Control, readmitted): difference = 40 - 30 = 10; squared = 100; term = 100/30. Cell (Control, not readmitted): difference = 60 - 70 = -10; squared = 100; term = 100/70.

• Sum all four terms to get the test statistic. The two 100/30 terms sum to 200/30, and the two 100/70 terms sum to 200/70. Converting to a common denominator of 210: 200/30 = 1400/210 and 200/70 = 600/210, so the total is 2000/210 = 9.524.

• Degrees of freedom for a 2x2 table = (2 minus 1) times (2 minus 1) = 1. With the test statistic equal to 9.524 and df = 1, the p-value is approximately 0.002. There is strong evidence against independence.

• Compute the risk difference and relative risk. Risk in Program group = 20 / 100 = 0.20. Risk in Control group = 40 / 100 = 0.40. Risk difference = 0.20 - 0.40 = -0.20 (the program group has 20 fewer events per 100). Relative risk = 0.20 / 0.40 = 0.50 (50% lower risk in the program group).

• Interpretation: the chi-square test confirms a statistically significant association (p approximately 0.002). The clinically meaningful summary is that the program is associated with a 20 percentage-point reduction in 30-day readmissions (RD = -0.20, RR = 0.50). The chi-square p-value alone does not convey this magnitude.

Result

Expected counts: E(Program readmitted) = 6000/200 = 30, E(Program not readmitted) = 14000/200 = 70, E(Control readmitted) = 6000/200 = 30, E(Control not readmitted) = 14000/200 = 70. Test statistic = 2000/210 = 9.524, df = 1, p approximately 0.002. Risk difference = 20/100 - 40/100 = -0.20 (20 fewer readmissions per 100 patients). Relative risk = (20/100) / (40/100) = 0.20 / 0.40 = 0.50. The effect size (RD, RR) is the deliverable; the p-value only confirms non-randomness.

Runnable example

import math
from scipy import stats
import numpy as np

# ── Motivating dataset: 30-day readmission (2x2 table) ──
# Rows: [Program, Control]; Columns: [Readmitted, Not readmitted]
table = np.array([[20, 80],
                  [40, 60]])

# ── 1. Chi-square test of independence ──
# correction=False: disable Yates' continuity correction (recommended when expected counts >= 5)
chi2, p_val, dof, expected = stats.chi2_contingency(table, correction=False)
print(f"Observed counts:\n{table}")
print(f"\nExpected counts:\n{expected}")
print(f"\nChi-square = {chi2:.4f}, df = {dof}, p = {p_val:.4f}")
print(f"Minimum expected count: {expected.min():.1f} (must be >= 5 for chi-square approximation)")

# ── 2. Fisher exact test (always valid; prefer when any expected count < 5) ──
_, fisher_p = stats.fisher_exact(table)
print(f"\nFisher exact p = {fisher_p:.4f}")

# ── 3. Effect sizes: risk difference and relative risk ──
n_a = table[0].sum()          # 100 patients in Program group
n_b = table[1].sum()          # 100 patients in Control group
risk_a = table[0, 0] / n_a   # 20/100 = 0.20
risk_b = table[1, 0] / n_b   # 40/100 = 0.40
rd = risk_a - risk_b          # risk difference: -0.20
rr = risk_a / risk_b          # relative risk: 0.50
print(f"\nRisk in Program group:  {risk_a:.3f}")
print(f"Risk in Control group:  {risk_b:.3f}")
print(f"Risk difference (A-B):  {rd:.3f}  (i.e., {rd*100:.0f} fewer events per 100 patients)")
print(f"Relative risk (A/B):    {rr:.3f}")

# 95% CI for risk difference (Newcombe method via normal approximation here for brevity)
se_rd = math.sqrt(risk_a * (1 - risk_a) / n_a + risk_b * (1 - risk_b) / n_b)
rd_lower, rd_upper = rd - 1.96 * se_rd, rd + 1.96 * se_rd
print(f"95% CI for RD: ({rd_lower:.3f}, {rd_upper:.3f})")

# ── 4. Cramér's V: standardized chi-square effect size ──
n_total = table.sum()
cramers_v = math.sqrt(chi2 / (n_total * (min(table.shape) - 1)))
print(f"\nCramér's V = {cramers_v:.3f}  (0=no association, 1=perfect association)")

# ── 5. Rule: check expected counts before deciding which test to report ──
if expected.min() < 5:
    print("\nWARNING: Expected count < 5. Report Fisher exact p-value, not chi-square.")
else:
    print("\nAll expected counts >= 5: chi-square approximation is valid.")

# ── Motivating dataset: 30-day readmission (2x2 table) ──
tab <- matrix(c(20, 80, 40, 60), nrow = 2, byrow = TRUE,
              dimnames = list(c("Program","Control"), c("Readmitted","Not_Readmitted")))
cat("Observed counts:\n"); print(tab)

# ── 1. Chi-square test of independence ──
# correct = FALSE: disable Yates' continuity correction (recommended when expected >= 5)
chi_res <- chisq.test(tab, correct = FALSE)
cat("\nChi-square result:\n"); print(chi_res)
cat("\nExpected counts:\n"); print(chi_res$expected)
cat(sprintf("Minimum expected count: %.1f\n", min(chi_res$expected)))

# ── 2. Fisher exact test (exact p-value; always valid for 2x2 tables) ──
fish_res <- fisher.test(tab)
cat(sprintf("\nFisher exact p = %.4f  OR = %.3f\n",
            fish_res$p.value, fish_res$estimate))

# ── 3. Effect sizes: risk difference and relative risk ──
risk_a <- tab["Program","Readmitted"] / sum(tab["Program",])    # 20/100
risk_b <- tab["Control","Readmitted"] / sum(tab["Control",])    # 40/100
rd     <- risk_a - risk_b   # -0.20
rr     <- risk_a / risk_b   # 0.50
cat(sprintf("\nRisk (Program) = %.3f, Risk (Control) = %.3f\n", risk_a, risk_b))
cat(sprintf("Risk difference = %.3f  (%.0f fewer events per 100 patients)\n", rd, rd * 100))
cat(sprintf("Relative risk   = %.3f\n", rr))

# Simple 95% CI for risk difference (normal approximation)
n_a <- sum(tab["Program",]); n_b <- sum(tab["Control",])
se_rd <- sqrt(risk_a * (1 - risk_a) / n_a + risk_b * (1 - risk_b) / n_b)
cat(sprintf("95%% CI for RD: (%.3f, %.3f)\n", rd - 1.96 * se_rd, rd + 1.96 * se_rd))

# ── 4. Cramér's V ──
n_total  <- sum(tab)
chi2_val <- chi_res$statistic
cramers_v <- sqrt(chi2_val / (n_total * (min(dim(tab)) - 1)))
cat(sprintf("\nCramér's V = %.3f\n", cramers_v))

# ── 5. Guard: switch to Fisher exact when expected counts < 5 ──
if (min(chi_res$expected) < 5) {
  cat("WARNING: Expected count < 5. Use Fisher exact p-value instead of chi-square.\n")
} else {
  cat("All expected counts >= 5: chi-square approximation valid.\n")
}

# ── 6. Cochran-Armitage trend test for ordered categories (example) ──
# For ordered dose groups (0,1,2,3) and a binary outcome:
# prop.trend.test(x = c(events_per_group), n = c(total_per_group))
# Example: event counts and group sizes across 4 ordered dose levels
events <- c(5, 10, 18, 30)
totals <- c(50, 50, 50, 50)
ca_res <- prop.trend.test(events, totals)
cat("\nCochran-Armitage trend test (4 ordered dose groups):\n"); print(ca_res)