Chi-Square Test of Independence

What the chi-square test of independence actually does

Given a contingency table with r rows (groups) and c columns (outcome categories), the test compares the observed count in every cell to the count that would be expected if the row variable and the column variable were statistically independent. The expected count for cell (i, j) is:

E(i,j) = (row i total × column j total) / grand total

The Pearson chi-square statistic sums the squared, scaled discrepancies across all cells:

χ² = Σ [ (O - E)² / E ]

Under the null hypothesis of independence this statistic follows (approximately) a chi-squared distribution with degrees of freedom (r − 1)(c − 1). A large χ² signals that at least one cell count is further from its expected value than pure chance would produce. The p-value is the probability of observing a χ² at least this large if independence were truly the case.

The chi-squared approximation is valid when the expected counts are large enough for the Central Limit Theorem to have kicked in. The widely cited rule of thumb — every expected cell count should be ≥ 5 — is a practical heuristic, not a theorem. Campbell (2007) recommends the more precise criterion that no more than 20 % of expected counts fall below 5 and none falls below 1. When this criterion is violated, Fisher's exact test is preferred for 2 × 2 tables; for larger tables, the options include collapsing sparse categories, a mid-p correction, or exact methods in software.

Degrees of freedom and table structure

For a 2 × 2 table (two groups, binary outcome), df = 1. For a 3 × 2 table (three groups, binary outcome), df = 2. For a 4 × 3 table, df = 6. The degrees of freedom capture how many cells are free to vary once the row and column totals are fixed. Increasing df reduces the chance of a large χ² under the null, so a given χ² value corresponds to a larger p-value as the table grows.

Yates' continuity correction — the debate

For 2 × 2 tables specifically, Yates (1934) proposed subtracting 0.5 from each |O − E| before squaring, bringing the discrete chi-square distribution closer to the continuous chi-squared curve. The Yates correction makes the test more conservative (larger p-values). The modern consensus, reviewed by Serra and Rea (2022), is that the correction is over-conservative: it produces p-values larger than the true exact p-value from Fisher's exact test, and Fisher's exact test itself is the preferred alternative when expected counts are small. Most analysts therefore do not apply Yates' correction as a default; when expected counts are small, they switch to Fisher's exact test entirely. In software, the `correction=False` flag in `scipy.stats.chi2_contingency` and `correct=FALSE` in `chisq.test` disable Yates' correction and are the appropriate defaults when expected counts are adequate.

Chi-square for trend — Cochran-Armitage

When the column categories are ordered (e.g., dose groups: none, low, medium, high), the standard chi-square test discards that ordering and tests only whether any pattern of association exists. The Cochran-Armitage trend test uses the ordering explicitly to construct a more powerful one-degree-of-freedom test for a monotone dose-response relationship. In pharmacoepidemiology and benefit-risk assessment, where ordered exposure categories are common, the trend test is often more appropriate than a general chi-square; it also produces a cleaner narrative — "risk increases with dose" — rather than the omnibus "at least one group differs."

Effect sizes: the deliverable that chi-square alone cannot provide

The chi-square statistic and its p-value answer only one question: is there evidence against independence? They do not tell you whether the association is clinically or economically meaningful. For binary outcomes in a 2 × 2 table, the effect size choices are:

• Risk difference (RD): proportion with event in group A minus proportion in group B. Directly interpretable: "8 fewer events per 100 patients." Sensitive to baseline risk.
• Relative risk (RR): event rate in group A divided by event rate in group B. Intuitive multiplicative framing: "40 % lower risk." Preferred in clinical communication and in prospective or cohort designs.
• Odds ratio (OR): the odds of the event in A divided by odds in B. Approximates the RR when the outcome is rare (< 10 %); diverges substantially for common outcomes. Required output of logistic regression; used extensively in case-control studies.
• Phi coefficient / Cramér's V: standardized chi-square-based effect size. Phi applies to 2 × 2 tables; Cramér's V generalises to larger tables. Values near 0 indicate weak association; near 1 indicate strong. Ben-Shachar et al. (2023) provide a modern treatment of these statistics. Phi and Cramér's V are most useful for comparing association strength across tables of different sizes, not for clinical communication.

For HEOR and HTA submissions, risk difference and relative risk (with 95 % confidence intervals) are the effect measures that connect most directly to budget-impact models, absolute risk reduction, and number-needed-to-treat calculations. A chi-square p-value without these effect estimates is insufficient for a decision-making audience.

RWE realities: the large-n problem and why p-values mislead at claims scale

In real-world evidence studies using administrative claims or EHR data, sample sizes of 50,000 to several million patients are routine. At this scale, the chi-square test will reject the null for association differences that are substantively meaningless. A 0.2 percentage-point difference in comorbidity prevalence between two cohorts of 500,000 patients will yield χ² values in the hundreds and p-values indistinguishable from zero. This is not an error in the test — it is working exactly as designed. The test is highly powered to detect any departure from independence, no matter how trivial.

The practical implication is that in large observational datasets, p-values from chi-square tests should be treated as nuisance outputs, not deliverables. What matters is:

1. The absolute risk difference and its 95 % confidence interval, sized against a pre-specified minimally important difference.
2. The standardized difference (also called the standardized mean difference for binary variables, where it equals the risk difference divided by the pooled standard deviation of the proportion). A standardized difference < 0.10 indicates negligible imbalance for covariate balance assessment, regardless of n.

For Table 1 balance comparisons in observational RWE studies, standardized differences are now the preferred summary because, unlike chi-square p-values, they do not inflate with sample size. Reporting chi-square p-values in a Table 1 comparing a treated group of 200,000 to a control group of 200,000 will produce a list of "statistically significant" differences in 80 %+ of covariates even after careful propensity-score matching — this is uninformative and potentially misleading.

Independence assumption violations in real-world data

The chi-square test assumes that each observation (each row in the contingency table) is independent. This assumption is violated in several common RWE settings:

• Claim-level rather than patient-level analysis: a patient with 12 hospitalizations contributes 12 rows. If the analyst builds a contingency table from claim rows rather than from patient-level aggregates, the rows are correlated within patient. Naive chi-square on such data is wrong — the effective sample size is far smaller than the row count, and standard errors are underestimated. The correct unit of analysis is almost always the patient; aggregate to one row per patient before any test.
• Clustering within providers or facilities: patients treated by the same physician or at the same hospital share outcomes that are more similar than random. If the chi-square table is built from cross-sectional data with this structure, a cluster-robust or mixed-model approach is needed; the standard test is anti-conservative.
• Repeated binary outcomes per patient over time: if a patient can have the outcome in multiple follow-up windows, standard chi-square treats each occurrence independently. Use GEE (generalized estimating equations) with an exchangeable working correlation structure, or collapse to a time-to-first-event analysis.

Pros, cons, and trade-offs

Pros:

• Conceptually simple: observed vs expected, with a single formula most analysts can verify by hand for a 2 × 2 table.
• Valid under minimal assumptions when expected counts are adequate (≥ 5 in each cell).
• Available in every statistical package; the implementation is identical across software.
• Extends naturally to r × c tables with a single statistic and a single p-value.
• At large n, the approximation is excellent; the test is computationally trivial.
• The Cochran-Armitage extension handles ordered categorical predictors without additional modelling machinery.

Cons:

• Does not produce a clinically meaningful effect estimate; risk difference, RR, or OR must be computed separately and are the actual deliverable for decision-makers.
• The chi-squared approximation breaks down when expected counts are small — the p-value is unreliable in sparse tables.
• Completely powered at large n: trivial associations are flagged as statistically significant; p-values alone are uninterpretable at claims scale.
• Cannot accommodate covariates or confounding adjustment; it is a marginal (unadjusted) test. For adjusted inference, route to logistic regression or log-binomial regression.
• Does not distinguish the direction of the association — a large χ² could reflect excess events in group A or in group B; the contingency table must be inspected.
• Sensitive to the unit of analysis: claim-level rows rather than patient-level rows produce an anti-conservative test.

Trade-offs vs alternatives:

• Fisher's exact test: exact p-value from the hypergeometric distribution; no large-sample approximation needed; preferred when expected counts are small; computationally heavier for large tables (though modern software handles this for typical RWE table sizes).
• McNemar's test: the paired-data analogue for a 2 × 2 table where the two rows represent the same subjects measured twice (e.g., pre/post or matched pairs). Using a standard chi-square test on paired binary data is a common and consequential mistake.
• Logistic regression: extends chi-square from a bivariate marginal test to a multivariable adjusted model; produces an OR with CI; handles continuous covariates; is the adjusted version of the chi-square test and the preferred primary method for binary outcomes in confounded observational comparisons.

When NOT to use — and when it is actively misleading

• Paired data (pre-post or matched pairs): if the two rows of the table represent the same patients before and after (or matched patient pairs), observations are not independent and chi-square is wrong. Use McNemar's test. This is one of the most common misapplications of chi-square in the biomedical literature.
• Small expected counts: when any expected cell count falls below 5 (or more than 20 % of cells are below 5), switch to Fisher's exact test for 2 × 2 tables. For larger tables, consider collapsing sparse categories or using an exact method.
• Claim-level rather than patient-level units: building the contingency table from claim rows creates correlated observations within patient; the effective sample size is overstated and the chi-square p-value is anti-conservative. Always aggregate to one row per patient before testing.
• As evidence of a causal association: chi-square tests for statistical association, not causation. An unadjusted chi-square on an unbalanced observational dataset estimates a confounded marginal association. Do not interpret a significant p-value as evidence that the exposure causes the outcome; route to propensity-score adjusted, regression-adjusted, or g-method analyses for causal inference.
• As the primary analysis for confounded comparisons: in any observational design without randomisation, the chi-square is at best a descriptive screen; the confirmatory analysis requires adjustment for confounders. Report the chi-square alongside the adjusted OR from logistic regression, not instead of it.
• At large n for substantive decision-making: at claims scale (hundreds of thousands of patients), every chi-square test will be significant. Report absolute risk differences, relative risks, standardized differences, and confidence intervals. Suppress or minimise the p-value, or at minimum note that at this sample size statistical significance is not informative.
• Ordered categories without exploiting the order: if the columns represent an ordered scale (severity grades, dose levels), the standard chi-square discards the ordering and is less powerful than the Cochran-Armitage trend test. Use the trend test for ordered alternatives.

Interpreting the output

In the worked example, a discharge coordination program is compared with usual care on 30-day readmission in a 2×2 contingency table. The Pearson chi-square statistic is approximately 9.52 on df = 1, with p ≈ 0.002. The estimated risk difference is −0.20 (20 fewer readmissions per 100 patients) and the relative risk is 0.50 (half the readmission rate in the program arm relative to usual care).

(1) Formal interpretation. The chi-square statistic of approximately 9.52 is the weighted sum of squared discrepancies between observed and expected cell counts. Under the null hypothesis of no association between program assignment and readmission, a statistic this large or larger on df = 1 arises by chance in approximately 0.2% of samples (p ≈ 0.002). The test confirms that the observed association is incompatible with independence at conventional significance levels. The p-value alone does not quantify the magnitude of the association — the risk difference (−0.20, i.e., 20 fewer readmissions per 100 patients) and the relative risk (0.50) are the effect measures that carry clinical and economic meaning.

(2) Practical interpretation. The data are consistent with the discharge program being associated with roughly half the 30-day readmission rate of usual care — a risk difference of 20 per 100 that is substantial in most hospital settings. The chi-square p-value confirms the association is unlikely to be pure chance; it does not establish causation. Because this is an unadjusted comparison, differences in baseline risk, patient mix, and facility factors between the two groups may confound the estimate. An adjusted logistic regression model using the same outcome is the appropriate next step for confirmatory inference.

Decision diagram

flowchart TD
  Start["Binary or multinomial categorical outcome?<br/>(group x outcome counts in a table)"] --> Paired{Paired data?<br/>Same patients appear<br/>in both rows}
  Paired -->|Yes| McNemar["Use McNemar's test<br/>(paired 2x2)<br/>NOT chi-square"]
  Paired -->|No| Unit{Unit of analysis<br/>correct?}
  Unit -->|Claim/visit rows<br/>per patient| Fix["Aggregate to<br/>one row per patient first"]
  Fix --> Unit
  Unit -->|One row<br/>per patient| ExpCount["Compute expected counts<br/>E = (row total × col total) / N"]
  ExpCount --> Check{Any expected<br/>count < 5?}
  Check -->|Yes| Fisher["Fisher exact test<br/>(2x2) or collapse<br/>sparse categories"]
  Check -->|No| ChiSq["Chi-square test<br/>χ² = Σ (O-E)² / E<br/>df = (r-1)(c-1)"]
  ChiSq --> Ordered{Columns ordered?<br/>dose levels,<br/>severity grades}
  Ordered -->|Yes| Trend["Cochran-Armitage<br/>trend test<br/>(more powerful)"]
  Ordered -->|No| Effect["Always report effect size:<br/>Risk difference, RR, OR<br/>NOT just the p-value"]
  Fisher --> Effect
  Trend --> Effect
  Effect --> LargeN{Large n?<br/>> 10,000 patients}
  LargeN -->|Yes| SMD["Report standardized differences<br/>and absolute risk reduction.<br/>p-value not informative at scale."]
  LargeN -->|No| Done["Report chi-square + p,<br/>effect size + 95% CI"]

Worked example

Scenario

A health outcomes analyst is comparing 30-day hospital readmission rates between patients who received a new discharge coordination program (Program group) and patients who received usual care (Control group). Each group has 100 patients. The analyst builds a 2x2 contingency table, computes expected counts, calculates the chi-square statistic by hand, and reports the risk difference and relative risk alongside the p-value.

Dataset

30-day readmission counts for 100 patients per group. Totals chosen so every expected count is a whole number and each (O-E)^2/E term is an exact fraction.

Steps

Result

Expected counts: E(Program readmitted) = 6000/200 = 30, E(Program not readmitted) = 14000/200 = 70, E(Control readmitted) = 6000/200 = 30, E(Control not readmitted) = 14000/200 = 70. Test statistic = 2000/210 = 9.524, df = 1, p approximately 0.002. Risk difference = 20/100 - 40/100 = -0.20 (20 fewer readmissions per 100 patients). Relative risk = (20/100) / (40/100) = 0.20 / 0.40 = 0.50. The effect size (RD, RR) is the deliverable; the p-value only confirms non-randomness.

Trade-offs

Runnable example

import math
from scipy import stats
import numpy as np

# ── Motivating dataset: 30-day readmission (2x2 table) ──
# Rows: [Program, Control]; Columns: [Readmitted, Not readmitted]
table = np.array([[20, 80],
                  [40, 60]])

# ── 1. Chi-square test of independence ──
# correction=False: disable Yates' continuity correction (recommended when expected counts >= 5)
chi2, p_val, dof, expected = stats.chi2_contingency(table, correction=False)
print(f"Observed counts:\n{table}")
print(f"\nExpected counts:\n{expected}")
print(f"\nChi-square = {chi2:.4f}, df = {dof}, p = {p_val:.4f}")
print(f"Minimum expected count: {expected.min():.1f} (must be >= 5 for chi-square approximation)")

# ── 2. Fisher exact test (always valid; prefer when any expected count < 5) ──
_, fisher_p = stats.fisher_exact(table)
print(f"\nFisher exact p = {fisher_p:.4f}")

# ── 3. Effect sizes: risk difference and relative risk ──
n_a = table[0].sum()          # 100 patients in Program group
n_b = table[1].sum()          # 100 patients in Control group
risk_a = table[0, 0] / n_a   # 20/100 = 0.20
risk_b = table[1, 0] / n_b   # 40/100 = 0.40
rd = risk_a - risk_b          # risk difference: -0.20
rr = risk_a / risk_b          # relative risk: 0.50
print(f"\nRisk in Program group:  {risk_a:.3f}")
print(f"Risk in Control group:  {risk_b:.3f}")
print(f"Risk difference (A-B):  {rd:.3f}  (i.e., {rd*100:.0f} fewer events per 100 patients)")
print(f"Relative risk (A/B):    {rr:.3f}")

# 95% CI for risk difference (Newcombe method via normal approximation here for brevity)
se_rd = math.sqrt(risk_a * (1 - risk_a) / n_a + risk_b * (1 - risk_b) / n_b)
rd_lower, rd_upper = rd - 1.96 * se_rd, rd + 1.96 * se_rd
print(f"95% CI for RD: ({rd_lower:.3f}, {rd_upper:.3f})")

# ── 4. Cramér's V: standardized chi-square effect size ──
n_total = table.sum()
cramers_v = math.sqrt(chi2 / (n_total * (min(table.shape) - 1)))
print(f"\nCramér's V = {cramers_v:.3f}  (0=no association, 1=perfect association)")

# ── 5. Rule: check expected counts before deciding which test to report ──
if expected.min() < 5:
    print("\nWARNING: Expected count < 5. Report Fisher exact p-value, not chi-square.")
else:
    print("\nAll expected counts >= 5: chi-square approximation is valid.")

# ── Motivating dataset: 30-day readmission (2x2 table) ──
tab <- matrix(c(20, 80, 40, 60), nrow = 2, byrow = TRUE,
              dimnames = list(c("Program","Control"), c("Readmitted","Not_Readmitted")))
cat("Observed counts:\n"); print(tab)

# ── 1. Chi-square test of independence ──
# correct = FALSE: disable Yates' continuity correction (recommended when expected >= 5)
chi_res <- chisq.test(tab, correct = FALSE)
cat("\nChi-square result:\n"); print(chi_res)
cat("\nExpected counts:\n"); print(chi_res$expected)
cat(sprintf("Minimum expected count: %.1f\n", min(chi_res$expected)))

# ── 2. Fisher exact test (exact p-value; always valid for 2x2 tables) ──
fish_res <- fisher.test(tab)
cat(sprintf("\nFisher exact p = %.4f  OR = %.3f\n",
            fish_res$p.value, fish_res$estimate))

# ── 3. Effect sizes: risk difference and relative risk ──
risk_a <- tab["Program","Readmitted"] / sum(tab["Program",])    # 20/100
risk_b <- tab["Control","Readmitted"] / sum(tab["Control",])    # 40/100
rd     <- risk_a - risk_b   # -0.20
rr     <- risk_a / risk_b   # 0.50
cat(sprintf("\nRisk (Program) = %.3f, Risk (Control) = %.3f\n", risk_a, risk_b))
cat(sprintf("Risk difference = %.3f  (%.0f fewer events per 100 patients)\n", rd, rd * 100))
cat(sprintf("Relative risk   = %.3f\n", rr))

# Simple 95% CI for risk difference (normal approximation)
n_a <- sum(tab["Program",]); n_b <- sum(tab["Control",])
se_rd <- sqrt(risk_a * (1 - risk_a) / n_a + risk_b * (1 - risk_b) / n_b)
cat(sprintf("95%% CI for RD: (%.3f, %.3f)\n", rd - 1.96 * se_rd, rd + 1.96 * se_rd))

# ── 4. Cramér's V ──
n_total  <- sum(tab)
chi2_val <- chi_res$statistic
cramers_v <- sqrt(chi2_val / (n_total * (min(dim(tab)) - 1)))
cat(sprintf("\nCramér's V = %.3f\n", cramers_v))

# ── 5. Guard: switch to Fisher exact when expected counts < 5 ──
if (min(chi_res$expected) < 5) {
  cat("WARNING: Expected count < 5. Use Fisher exact p-value instead of chi-square.\n")
} else {
  cat("All expected counts >= 5: chi-square approximation valid.\n")
}

# ── 6. Cochran-Armitage trend test for ordered categories (example) ──
# For ordered dose groups (0,1,2,3) and a binary outcome:
# prop.trend.test(x = c(events_per_group), n = c(total_per_group))
# Example: event counts and group sizes across 4 ordered dose levels
events <- c(5, 10, 18, 30)
totals <- c(50, 50, 50, 50)
ca_res <- prop.trend.test(events, totals)
cat("\nCochran-Armitage trend test (4 ordered dose groups):\n"); print(ca_res)

/* ── Create motivating dataset ── */
data work.readmit;
  input group $ outcome $ count;
  datalines;
Program Readmitted    20
Program Not_Readmitted 80
Control Readmitted    40
Control Not_Readmitted 60
;
run;

/* ── 1. Chi-square test, Fisher exact, expected counts, risk difference, relative risk ── */
proc freq data=work.readmit;
  weight count;
  tables group * outcome /
    chisq          /* Pearson chi-square (and Yates-corrected version in output)          */
    fisher         /* Fisher exact test (exact p-value; prefer when expected count < 5)   */
    expected       /* Print expected cell frequencies to verify >= 5 rule                 */
    nocol norow    /* Suppress row/column % (keep cell % and counts cleaner)              */
    riskdiff       /* Risk difference (row 1 vs row 2) with 95% CI                        */
    relrisk;       /* Relative risk and odds ratio with 95% CI                            */
  /* Note: SAS prints BOTH the uncorrected and Yates-corrected chi-square.                */
  /* For adequate expected counts (>= 5), use the uncorrected Pearson chi-square row.     */
  /* When any expected count < 5, use the Fisher exact p-value in the output.             */
run;

/* ── 2. Manual verification: expected counts via PROC FREQ output data set ── */
proc freq data=work.readmit noprint;
  weight count;
  tables group * outcome / expected out=work.freq_out outexpected;
run;
proc print data=work.freq_out; run;

/* ── 3. Cochran-Armitage trend test for ordered categories ── */
/* Example: event counts across 4 ordered dose levels (1=low, 4=high)           */
data work.trend;
  input dose events total;
  datalines;
1 5  50
2 10 50
3 18 50
4 30 50
;
run;
/* Expand to individual rows for PROC FREQ (binary outcome per patient) */
data work.trend_expanded;
  set work.trend;
  do i = 1 to events;   outcome = 1; output; end;
  do i = 1 to (total - events); outcome = 0; output; end;
  drop i;
run;
proc freq data=work.trend_expanded;
  tables dose * outcome / trend;  /* TREND option: Cochran-Armitage trend test */
run;

Citations

1. [1]Campbell I. Chi-squared and Fisher-Irwin tests of two-by-two tables with small sample recommendations. Statistics in Medicine. 2007;26(19):3661-3675.
2. [2]McHugh ML. The Chi-square test of independence. Biochemia Medica. 2013;23(2):143-149.

1. [3]Serra A, Rea F, Di Carlo A. Continuity correction of Pearson's chi-square test in 2x2 contingency tables: a mini-review on recent development. Epidemiology, Biostatistics, and Public Health. 2022;19(1):e17027.

1. [4]Ben-Shachar MS, Patil I, Thériault R, Wiernik BM. Phi, Fei, Fo, Fum: Effect sizes for categorical data that use the chi-squared statistic. Mathematics. 2023;11(9):1982.