Fisher's Exact Test

What Fisher's Exact Test does and why "exact" matters

Fisher's Exact Test answers one question: given that we observe a 2×2 table with fixed marginal totals, how often would we see this arrangement — or one even more lopsided — if there were truly no association between the row and column variables?

The test works by conditioning on the marginal totals (both row totals and both column totals are treated as fixed), which reduces the problem to a single hypergeometric distribution. Under the null hypothesis of no association, the upper-left cell count follows a hypergeometric distribution with parameters determined by the four marginals. The exact p-value is the sum of hypergeometric probabilities for all tables with those same marginals that are at least as extreme (have upper-left cell count as small or smaller) as the observed table.

Because the p-value is computed directly from the hypergeometric PMF rather than from a chi-square approximation, it is "exact" in the sense that it does not depend on an asymptotic argument. The type-I error rate is guaranteed not to exceed the nominal alpha for any table size — including tables with n = 10 patients or with one or more cells containing zeros. This guarantee is precisely what Fisher's test offers that the Pearson chi-square test cannot.

The hypergeometric probability in detail

A 2×2 table has four cells denoted as follows: a (row 1 / col 1), b (row 1 / col 2), c (row 2 / col 1), d (row 2 / col 2). The row totals are R1 = a + b and R2 = c + d; the column totals are C1 = a + c and C2 = b + d; the grand total is N = R1 + R2.

Under the null hypothesis of no association, with all margins fixed, the probability of observing exactly a events in the upper-left cell is the hypergeometric PMF:

P(X = a) = C(R1, a) × C(R2, C1 - a) / C(N, C1)

where C(n, k) = n! / [k! × (n-k)!] is the binomial coefficient. The p-value for a one-sided test (testing whether the upper-left cell count is smaller than expected) is the sum of this probability over all values of a from 0 up to the observed value; the standard two-sided p-value doubles this sum (or, more precisely, sums over all tables whose hypergeometric probability does not exceed that of the observed table).

Why conditioning on margins is controversial

Ronald Fisher's original justification for conditioning on both margins rested on the "Lady Tasting Tea" experiment, where both the row and column totals are genuinely fixed by design. In most medical and epidemiological studies this is not the case: only the row totals (sample sizes per group) are fixed, not the column totals (event counts). When only one margin is fixed, conditioning on the other margin discards information about the population proportion of events — it treats a quantity that could distinguish the null from the alternative as nuisance.

This has a practical consequence: Fisher's test is conservative — its actual type-I error rate is strictly below the nominal alpha for most discrete tables, because the hypergeometric distribution is discrete and the exact tail probability rarely hits alpha exactly. Conservatism means lower power than you could have if you used an unconditional exact test (Barnard's test) or the mid-p correction.

The conservatism critique and the mid-p alternative

The conservatism of Fisher's test is well-established in the statistical literature. Lydersen, Fagerland, and Laake (2009) conducted a comprehensive simulation comparing eighteen tests for 2×2 tables and reached the conclusion that Fisher's test is often not the best choice: its actual size is far below the nominal level in many scenarios (sometimes less than half of alpha), leading to a systematic loss of power that is particularly damaging in small studies where power is already scarce.

The mid-p correction addresses this by computing:

mid-p = (1/2) × P(X = a_observed) + P(X < a_observed)

This is not a valid frequentist p-value in the strict sense (it does not guarantee conservative type-I error control), but it brings the actual size much closer to the nominal level while remaining more conservative than the chi-square approximation. Lydersen et al. (2009) recommend the mid-p correction as a practical improvement over the standard Fisher p-value for most small-sample 2×2 tables. Unconditional exact tests (Barnard's test, Boschloo's test) are theoretically superior in terms of power, but computationally heavier and not universally implemented.

The zero-cell problem and its consequences for odds ratios

In sparse safety tables — a rare adverse event occurring in 0 of 10 exposed patients versus 0 of 50 controls, or 1 of 100 versus 0 of 200 — the conditional odds ratio (OR) is undefined: zero cells make the cross-product ratio 0/0 or a/0. Fisher's test can still produce a valid p-value in these situations because the hypergeometric calculation does not require the OR to be defined, but it cannot produce a finite OR or OR-based confidence interval.

Three common remedies for the zero-cell OR problem:

1. Continuity correction (add 0.5 to each cell): produces a finite OR but the choice of 0.5 is arbitrary and the resulting CI is not exact.
2. Exact conditional CI (computed from the hypergeometric distribution): available from fisher.test in R and PROC FREQ EXACT FISHER in SAS; this CI is exact in the conditional sense but inherits Fisher's conservatism.
3. Firth penalized logistic regression: the modern recommended approach for sparse or zero-cell tables when an adjusted OR is needed. Firth's method adds a Jeffreys-prior penalty to the log-likelihood, which removes the separation problem and produces finite point estimates and profile likelihood CIs without any ad-hoc correction. When the analysis must adjust for covariates (as is nearly always the case in observational RWE), exact logistic regression or Firth logistic regression is the appropriate next step.

RWE and safety surveillance applications

Fisher's Exact Test appears in RWE most often in three contexts:

Spontaneous pharmacovigilance and signal tables: Disproportionality tables in regulatory safety databases often involve rare suspected adverse drug reactions; the MedDRA preferred term × drug pairing table may have single-digit cell counts even in large spontaneous reporting databases. Fisher's test is one of several signal-detection metrics, alongside the reporting odds ratio and proportional reporting ratio.

Small clinical cohorts and registry subgroups: A pragmatic trial or registry study in a rare disease may enroll 20 to 50 patients per arm. Serious adverse events in these cohorts produce 2×2 tables with expected counts below 5, violating the chi-square approximation. Fisher's test or the mid-p correction is appropriate here, with the caveat that power is limited and the absence of a statistically significant finding cannot be interpreted as proof of safety.

Post-market pharmacoepidemiology safety tables: Early post-approval surveillance of a newly approved drug may cover only a few thousand exposed patients. If the event of interest is rare (background rate < 1 per 1,000), observed counts may be 0, 1, or 2 even in large studies. Fisher's test handles these counts; TreeScan and maxSPRT add sequential analysis for repeated looks.

Scale point: when Fisher's test is computationally unnecessary

In large administrative claims databases — 100,000 to 10,000,000 person-years of follow-up — expected cell counts for even moderately rare events (incidence rate 0.1 per 100) will be in the hundreds or thousands per cell. At that scale the chi-square approximation is excellent, Fisher's test offers no advantage, and the computational cost (combinatorial for large N) is unnecessary. Fisher's test is the right tool when expected counts are small, not when the database is large. An analyst working in Medicare claims with 500,000 exposed patients comparing a cardiovascular event rate of 5% does not need Fisher's test.

Pros, cons, and trade-offs

Pros:

• Exact: the p-value is guaranteed to not exceed alpha under the null regardless of table size; valid for any n, including n = 5 or n = 0 in a cell.
• Universally implemented: scipy.stats.fisher_exact, R's fisher.test, SAS PROC FREQ EXACT FISHER are available in all analytical pipelines without additional packages.
• Handles zero cells: produces a valid p-value even when one cell contains zero events, unlike chi-square which returns NaN or requires continuity correction.
• Provides exact conditional CI for the OR: fisher.test in R returns the conditional maximum-likelihood OR estimate and an exact CI alongside the p-value.

Cons:

• Conservative: actual type-I error is below nominal alpha due to discreteness; power is lower than unconditional tests or mid-p correction in the same scenarios.
• Does not directly estimate an effect size: the test produces a p-value; the conditional OR and CI are a separate output and are undefined when cells contain zeros.
• Conditions on both margins: theoretically appropriate only when both margins are fixed by design; for observational 2×2 tables this assumption is debatable and leads to the conservatism described above.
• Computationally expensive for large N: the exact calculation becomes slow for tables with very large marginal totals, though this is rarely a practical problem because Fisher's test is only needed when N is small.
• Cannot adjust for confounders: like chi-square, Fisher's test produces an unadjusted association measure; when confounders are present, route to exact logistic regression or Firth logistic regression.

When NOT to use

Do not use Fisher's Exact Test when:

• All expected cell counts are ≥ 5 and n per group is adequate: chi-square is the correct choice; Fisher's conservatism wastes power unnecessarily.
• The design is paired: a 2×2 table comparing pre-post classifications, matched pairs, or before-after assessments on the same patients requires McNemar's test (or its exact variant), not Fisher's test. Applying Fisher's test to paired data ignores the within-patient correlation and produces an anti-conservative estimate.
• An adjusted odds ratio is needed: Fisher's test is an unadjusted marginal test; for observational data with confounders, route to logistic regression, and when cell counts are sparse or zero-cells are present, use exact logistic regression or Firth penalized logistic regression.
• The table is larger than 2×2 with ordered categories: ordinal × ordinal associations are better assessed with the Mantel-Haenszel trend test or ordinal logistic regression; Fisher's test for r×c tables loses information about the ordering.
• The study is large (n >> 1,000 per group) with common events: at this scale chi-square is fine; Fisher's test provides no accuracy benefit and misleads by association with "small-sample rigor" in a context where it is not needed.
• A relative risk or rate difference is the target estimand: Fisher's test is built around the conditional OR; for cohort studies where RR or RD is more directly interpretable, compute the RR/RD directly with a confidence interval.

Implementation note across languages

All three implementations below show the same 2×2 safety table (3 adverse events in 10 exposed vs 0 in 10 unexposed), compute the exact p-value, report the conditional OR and CI, and apply the mid-p correction by hand. This mirrors the worked example in the beginner layer so the analyst can trace from manual arithmetic to production code.

Interpreting the output

In the worked example, 3 of 10 patients in the drug arm experienced a rash and 0 of 10 in the placebo arm did. The one-sided exact p-value — the hypergeometric probability of observing 3 or more events in the exposed arm given fixed marginals of 10 exposed, 10 unexposed, and 3 total events — is 2/19 (approximately 0.105). The two-sided p-value is 4/19 (approximately 0.211). The mid-p one-sided value, which halves the probability of the observed table before accumulating the tail, is approximately 0.053.

(1) Formal interpretation. Under the null hypothesis of no association and conditioning on both sets of marginal totals, the exact probability of an arrangement as extreme or more extreme than the one observed is approximately 0.11 (one-sided) or 0.21 (two-sided). Neither exceeds the conventional alpha = 0.05 threshold, and the test does not reject the null. The mid-p correction brings the one-sided value to approximately 0.053 — still marginally above 0.05. Fisher's conservatism (which arises from conditioning on both margins) makes it the most cautious of the available exact tests; the non-rejection does not rule out a real adverse-event signal — it reflects the severe limitation of the sample size.

(2) Practical interpretation. Three rash events in 10 exposed patients versus zero in 10 unexposed patients is a directionally concerning imbalance that does not achieve conventional statistical significance in a study of 20 total patients. Fisher's exact test is the appropriate choice here because expected cell counts are well below 5, making the chi-square approximation unreliable. A non-significant result in a study this small cannot be interpreted as evidence of no rash risk — the study simply lacks the power to detect or rule out signals of this magnitude. Safety surveillance for rare adverse events requires much larger exposed populations and sequential monitoring methods before a signal can be confirmed or excluded.

Decision diagram

flowchart TD
  A[2x2 contingency table] --> B{All expected\ncounts >= 5?}
  B -- Yes --> C[Chi-square test\nwith effect estimate\nRD or RR or OR plus CI]
  B -- No --> D{Any cell = 0?}
  D -- No --> E[Fisher exact test\ntwo-sided\nplus exact conditional OR plus CI]
  D -- Yes --> F{Need adjusted\nestimate?}
  E --> G{Power a concern?}
  G -- Yes --> H[Consider mid-p correction\nas secondary analysis]
  G -- No --> I[Report Fisher p-value\nplus OR plus exact CI]
  F -- No --> J[Fisher exact test\nreport one-sided CI as bound\nnote OR undefined]
  F -- Yes --> K[Firth penalized\nlogistic regression\nor exact logistic]
  K --> L[Finite adjusted OR\nprofile likelihood CI\nconfounding controlled]

Worked example

Scenario

A safety monitor is reviewing a phase II trial in a rare disease. Ten patients received the new drug and ten received placebo. Three patients in the drug arm experienced a serious rash; zero patients in the placebo arm did. The expected counts are too small for chi-square (expected events in placebo cell = 1.5), so the monitor computes Fisher's Exact Test by hand to decide whether the imbalance is unlikely under the null hypothesis of no drug effect on rash.

Dataset

2x2 adverse-event table. Rows = treatment arm; columns = rash (yes/no). Row totals and column totals (margins) are used as the fixed inputs to Fisher's calculation.

Steps

Result

C(20,3) = 1140. P(X=0) = 120/1140, P(X=1) = 450/1140, P(X=2) = 450/1140, P(X=3) = 120/1140. Sum = 1140/1140 = 1. One-sided Fisher p-value = P(X=3) = 120/1140 = 2/19 which equals approximately 0.1053. Two-sided p = 240/1140 = 4/19 which equals approximately 0.2105. Mid-p one-sided = 60/1140 = 1/19 which equals approximately 0.0526. With only 10 patients per arm the test has very low power; neither the exact p-value nor the mid-p reaches conventional significance after two-sided correction, illustrating why rare-event detection in small cohorts requires sequential designs or pooled post-market data rather than a single-study test.

Trade-offs

Runnable example

from scipy import stats

# ── Safety table: drug vs placebo, rash yes/no ──────────────────────────────────
# Rows: [Drug, Placebo]; Columns: [Rash, No_Rash]
table = [[3, 7],   # Drug:    3 rash, 7 no rash
         [0, 10]]  # Placebo: 0 rash, 10 no rash

# ── 1. Standard two-sided Fisher exact test ──────────────────────────────────────
odds_ratio, p_two_sided = stats.fisher_exact(table, alternative="two-sided")
print(f"Odds ratio (cross-product):   {odds_ratio:.3f}")
print(f"Two-sided exact p-value:       {p_two_sided:.4f}")

# One-sided: testing whether drug arm has HIGHER rash rate
_, p_one_sided = stats.fisher_exact(table, alternative="greater")
print(f"One-sided exact p-value:       {p_one_sided:.4f}")

# ── 2. Mid-p correction (manual, from hypergeometric PMF) ─────────────────────────
# Under the null: X ~ Hypergeometric(N=20, K=3 rash events total, n=10 drug arm)
# scipy.stats.hypergeom(M, n, N): M=population, n=success states, N=draws
N_total = 20   # grand total
K_events = 3   # total rash events (column 1 total)
n_drug = 10    # drug arm size (row 1 total)
a_obs = 3      # observed drug-arm rash count

rv = stats.hypergeom(N_total, K_events, n_drug)
p_at_obs = rv.pmf(a_obs)          # P(X = 3)
p_more_extreme = rv.sf(a_obs)     # P(X > 3) = 0 (max is 3)
mid_p_one_sided = 0.5 * p_at_obs + p_more_extreme
print(f"\nMid-p one-sided:               {mid_p_one_sided:.4f}")
print(f"P(X=3) from hypergeom PMF:     {p_at_obs:.4f}")

# ── 3. Hypergeometric PMF for all possible a values ──────────────────────────────
print("\nHypergeometric probability table:")
for a in range(0, K_events + 1):
    p = rv.pmf(a)
    print(f"  P(X={a}) = C(10,{a}) x C(10,{K_events-a}) / C(20,{K_events}) = {p:.6f}")

# ── 4. Scale check: chi-square on a larger table ──────────────────────────────────
# In claims with 500 drug / 500 placebo and 5% event rate, chi-square is fine
large_table = [[25, 475], [15, 485]]  # 5% vs 3% events, n=500 per arm
chi2, chi2_p, dof, expected = stats.chi2_contingency(large_table, correction=False)
_, fisher_p_large = stats.fisher_exact(large_table)
print(f"\nLarge-sample (n=500/arm):")
print(f"  Expected min cell:  {expected.min():.1f}  (>> 5: chi-square valid)")
print(f"  Chi-square p:       {chi2_p:.4f}")
print(f"  Fisher exact p:     {fisher_p_large:.4f}  (nearly identical; chi-square sufficient)")

# ── Safety table: drug vs placebo, rash yes/no ──────────────────────────────────
tab <- matrix(c(3, 0, 7, 10), nrow = 2,
              dimnames = list(Arm = c("Drug", "Placebo"),
                              Rash = c("Yes", "No")))
cat("Observed table:\n"); print(tab)
cat("Expected counts (under null):\n")
print(chisq.test(tab, correct = FALSE)$expected)

# ── 1. Two-sided Fisher exact (standard clinical report) ──────────────────────────
ft2 <- fisher.test(tab, alternative = "two.sided")
cat("\nTwo-sided Fisher exact test:\n")
print(ft2)
# NOTE: ft2$estimate is the conditional MLE OR (differs from cross-product at small n)
# ft2$conf.int is the exact conditional 95% CI for this OR

# ── 2. One-sided (testing drug > placebo for rash) ───────────────────────────────
ft1 <- fisher.test(tab, alternative = "greater")
cat("\nOne-sided Fisher exact (drug > placebo):\n")
cat(sprintf("  p-value: %.4f\n", ft1$p.value))

# ── 3. Mid-p correction (manual from dhyper) ─────────────────────────────────────
# X ~ Hypergeometric(m=3 total rash, n=17 total no-rash, k=10 drug arm)
# dhyper(x, m, n, k): P(X=x) where m = K_events, n = N-K_events, k = n_drug
m_events  <- 3   # total rash events = column 1 total
n_no_event <- 17  # total no-rash = column 2 total
k_drug    <- 10  # drug arm size
a_obs     <- 3   # observed drug-arm rash count

p_at_obs    <- dhyper(a_obs, m_events, n_no_event, k_drug)
p_more_extreme <- phyper(a_obs, m_events, n_no_event, k_drug, lower.tail = FALSE)
mid_p_one_sided <- 0.5 * p_at_obs + p_more_extreme
cat(sprintf("\nMid-p one-sided: %.4f (vs conservative Fisher one-sided: %.4f)\n",
            mid_p_one_sided, ft1$p.value))

# ── 4. PMF table for all possible a values ────────────────────────────────────────
cat("\nHypergeometric PMF for all a = 0..3:\n")
for (a in 0:3) {
  cat(sprintf("  P(X=%d) = %.6f\n", a, dhyper(a, m_events, n_no_event, k_drug)))
}

# ── 5. Scale point: large table where chi-square is adequate ─────────────────────
large_tab <- matrix(c(25, 15, 475, 485), nrow = 2)
exp_min <- min(chisq.test(large_tab, correct = FALSE)$expected)
chi_p   <- chisq.test(large_tab, correct = FALSE)$p.value
fish_p  <- fisher.test(large_tab)$p.value
cat(sprintf("\nLarge n=500/arm: expected min = %.1f; chi2 p = %.4f; Fisher p = %.4f\n",
            exp_min, chi_p, fish_p))
cat("All expected counts >> 5: chi-square is appropriate; Fisher adds nothing.\n")

/* ── Create the safety dataset ─────────────────────────────────────────────── */
data work.safety;
  input arm $ rash $ count;
  datalines;
Drug    Yes  3
Drug    No   7
Placebo Yes  0
Placebo No  10
;
run;

/* ── 1. PROC FREQ: chi-square, Fisher exact, expected counts, OR + exact CI ── */
proc freq data=work.safety order=data;
  weight count;
  tables arm * rash / chisq fisher expected relrisk;
  /* CHISQ:    Pearson chi-square and Yates continuity-corrected chi-square       */
  /* FISHER:   Fisher exact test (one-sided and two-sided p-values)               */
  /* EXPECTED: Prints expected cell frequencies — check all are >= 5 before       */
  /*           relying on chi-square; here expected(Placebo,Yes) = 1.5 -> FAIL    */
  /* RELRISK:  Risk ratio and OR with Wald CI (note: MLE OR undefined if a=0;     */
  /*           SAS will warn; use EXACT FISHER OR option for exact conditional CI) */
run;

/* ── 2. EXACT conditional OR and CI from PROC FREQ ─────────────────────────── */
proc freq data=work.safety order=data;
  weight count;
  tables arm * rash / fisher;
  exact fisher or;
  /* OR option with EXACT: outputs the conditional MLE OR and exact 95% CI        */
  /* This is the OR from the noncentral hypergeometric distribution, same as R's  */
  /* fisher.test()$estimate — not the cross-product OR                            */
run;

/* ── 3. Mid-p correction via the hypergeometric PMF ────────────────────────── */
/* SAS does not output mid-p directly; compute from the hypergeometric PMF       */
data work.midp;
  /* Hypergeometric: X ~ Hyper(N=20, K=3, n=10)                                 */
  N_total  = 20;   /* grand total                                                */
  K_events = 3;    /* total events (column 1 sum)                                */
  n_drug   = 10;   /* drug arm size (row 1 total)                                */
  a_obs    = 3;    /* observed drug-arm event count                              */

  /* P(X = a_obs) from SAS hypergeometric PMF function                           */
  p_at_obs      = pdf('Hyper', a_obs, N_total, K_events, n_drug);
  /* P(X > a_obs) = 1 - CDF(a_obs) = upper tail                                 */
  p_more_extreme = 1 - cdf('Hyper', a_obs, N_total, K_events, n_drug);
  mid_p_one_sided = 0.5 * p_at_obs + p_more_extreme;

  put 'P(X=3) = ' p_at_obs 6.4;
  put 'Mid-p one-sided = ' mid_p_one_sided 6.4;
run;

/* ── 4. Firth logistic regression: adjusted OR for sparse tables ─────────────  */
/* When a covariate (e.g., age, baseline severity) must be adjusted, and zero    */
/* cells preclude standard logistic regression, use FIRTH option in PROC LOGISTIC */
/* Here we add a simulated age covariate for illustration                        */
data work.safety_indiv;
  set work.safety;
  /* Expand count to individual records                                           */
  do i = 1 to count;
    rash_bin = (rash = 'Yes');
    arm_bin  = (arm = 'Drug');
    /* Simulate a confounder: mean age 55 for drug, 60 for placebo              */
    if arm = 'Drug'    then age = 55 + rannor(42) * 5;
    else                    age = 60 + rannor(42) * 5;
    output;
  end;
  drop i;
run;

proc logistic data=work.safety_indiv;
  model rash_bin(event='1') = arm_bin age / firth;
  /* FIRTH: Penalized likelihood (Jeffreys prior) — produces finite OR even      */
  /*        when separation is complete or quasi-complete (zero cells)            */
  /* Outputs: point estimate, profile likelihood CI (not Wald), p-value          */
run;

Citations

1. [1]Lydersen S, Fagerland MW, Laake P. Recommended tests for association in 2x2 tables. Statistics in Medicine. 2009;28(7):1159-1175.
2. [2]Campbell I. Chi-squared and Fisher-Irwin tests of two-by-two tables with small sample recommendations. Statistics in Medicine. 2007;26(19):3661-3675.

1. [3]Heinze G, Schemper M. A solution to the problem of separation in logistic regression. Statistics in Medicine. 2002;21(16):2409-2419.

1. [4]Firth D. Bias reduction of maximum likelihood estimates. Biometrika. 1993;80(1):27-38.