Signal Detection (Disproportionality Analysis)

In plain language

Signal detection in drug safety screening asks one question over a database of voluntary adverse-event reports: is a particular drug paired with a particular side effect reported far more often than you would expect by chance? The core tool is a simple 2x2 table that counts how often the drug and event appear together versus apart, then computes a ratio called the PRR or ROR to measure that excess. A high ratio is called a signal of disproportionate reporting, which is a flag for experts to investigate further, not proof that the drug caused the event.

Disproportionality analysis

is the workhorse of quantitative pharmacovigilance signal detection. It does not estimate a risk, a rate, or a causal effect. It asks a single screening question over a spontaneous-reporting system (SRS) such as FDA FAERS, WHO VigiBase, or EudraVigilance: is a particular drug-event combination reported disproportionately more often than would be expected if reporting of that event were independent of that drug? Every measure — the proportional reporting ratio (PRR), reporting odds ratio (ROR), the information component (IC/BCPNN), and the empirical-Bayes geometric mean (EBGM) from the Gamma-Poisson Shrinker (GPS) — is a function of the same 2×2 contingency table collapsed across the entire database: cell `a` = reports naming both the drug and the event, `b` = the drug with other events, `c` = the event with other drugs, `d` = all other reports.

Core conceptual distinction

. The estimand here is a reporting association, not an incidence association. There is no person-time and no unexposed denominator — only counts of reports — so PRR and ROR are ratios of reporting proportions, fundamentally different from a risk ratio or rate ratio estimated in a cohort. The frequentist measures (PRR, ROR with their χ² and confidence intervals) and the Bayesian shrinkage measures (IC from the BCPNN, EBGM from the GPS) answer the same question but differ in one decisive way: shrinkage estimators pull the disproportionality of low-count pairs back toward the null, so they do not fire on the unstable a=1 or a=2 cells that produce huge, meaningless PRRs. PRR and ROR are nearly numerically identical when the event is rare relative to the database (b≈a+b, d≈c+d); they diverge for common events, where the ROR (an odds ratio) is the better-behaved statistic and the one used by the Netherlands Pharmacovigilance Centre and EudraVigilance. A signal of disproportionate reporting (SDR) is a screening alert, not a confirmed adverse drug reaction — it is the input to clinical review, not the conclusion.

Pros, cons, and trade-offs

. - vs Bayesian shrinkage (IC/BCPNN, EBGM/GPS): Plain PRR/ROR are transparent, trivially computed, and the basis of the MHRA operational rule, but they are wildly unstable at small `a` and over-fire on rare pairs. Shrinkage estimators (the FDA's EBGM/EB05 and WHO-UMC's IC025) trade interpretability for stability and far better operating characteristics in sparse, high-multiplicity tables. Prefer PRR/ROR for a small, transparent, regulator-facing screen of one product; prefer EBGM or IC for routine data-mining of millions of drug-event cells. - vs traditional pharmacoepidemiologic designs (cohort, case-control, SCCS): Disproportionality is almost free, runs over the whole world's spontaneous reports, and detects the unexpected; but it cannot estimate magnitude, cannot establish causation, and is hostage to reporting biases. Prefer disproportionality to generate hypotheses; switch to a designed study (active-comparator new-user cohort, self-controlled case series, sequential active surveillance in claims) to test them. - vs self-controlled / sequential active surveillance (TreeScan, MaxSPRT in Sentinel): SRS disproportionality has no denominator and no temporality; claims/EHR sequential methods add real person-time and exposure timing but cost orders of magnitude more effort and only cover their own population. Prefer SRS mining for breadth and speed; prefer active surveillance once a candidate signal warrants confirmation in a population with a denominator.

When to use

. Routine, broad, hypothesis-generating safety surveillance over a spontaneous-reporting database; periodic safety-update report (PSUR/PBRER) screening; new-product launch monitoring; flagging unexpected drug-event pairs for clinical and epidemiologic follow-up; and as the first stage of a tiered pharmacovigilance system whose later stages are designed studies.

When NOT to use — and when it is actively misleading or dangerous

. - As evidence of causation or to quantify risk. A high PRR means an event is reported more with a drug — driven as easily by stimulated reporting after a media story, a litigation campaign, or a label warning (notoriety/Weber effect) as by true excess risk. Treating an SDR as a confirmed ADR, or back-calculating an incidence from report counts, is the classic and dangerous misuse. - At very low counts. With a<3, frequentist PRR/ROR are numerically unstable and confidence intervals are uninformative; MHRA explicitly gates on n≥3 and shrinkage estimators exist precisely to tame this. - When masking is plausible. A single very large signal within a drug class (or a single dominant event) inflates the comparator margins and can hide a genuine smaller signal — disproportionality run without stratification or subsetting can be falsely reassuring. - For within-class or dose-response comparisons the raw whole-database 2×2 cannot answer; these require stratified or regression-based (logistic/Poisson) disproportionality, or a proper study. - Across heterogeneous report sources without stratification. Pooling spontaneous, solicited (patient support program), and literature reports, or pooling across regions with different reporting cultures, confounds the reporting association — stratify (Mantel-Haenszel PRR) or restrict.

Data-source operational depth

. - Spontaneous-reporting systems (FAERS, VigiBase, EudraVigilance) — the primary substrate: The unit is the report (an ICSR), not the patient. There is no denominator: you only ever see a/(a+b+c+d), so no rate, risk, or absolute frequency can be computed — only disproportionality. Failure modes: (1) Duplicates — the same case submitted by patient, physician, and manufacturer inflates `a`; run deduplication (e.g., VigiBase's vigiMatch, or FAERS case-version/primaryid logic keeping the latest follow-up) before tabulating. (2) Notoriety / stimulated reporting — a label change, Dear-Doctor letter, or lawsuit causes a reporting spike unrelated to incidence; interpret SDRs against the reporting timeline. (3) Indication confounding / event masking — the event may track the underlying disease, and a dominant class signal can mask a smaller one; mitigate with stratification or comparator restriction. (4) Term granularity — MedDRA Preferred Terms vs grouped Standardised MedDRA Queries (SMQs) change the 2×2 materially; pre-specify the level. (5) Multiplicity — millions of drug-event cells are screened, so naive α=0.05 thresholds generate thousands of false alerts; this is the operational reason shrinkage (EB05≥2, IC025>0) and fixed criteria (PRR≥2 & χ²≥4 & n≥3) replace p-values. - Claims / EHR (active surveillance extension): Disproportionality is not the natural method here — these data have denominators, so cohort, self-controlled, or sequential-testing methods (TreeScan tree-based scan statistics, MaxSPRT in FDA Sentinel) are used instead and constitute a different method family. If a disproportionality-style screen is run over claims, beware prevalent-user contamination, exposure misclassification from `days_supply` gaps, and look-elsewhere/multiplicity from scanning the full MedDRA/ICD tree. Do not present claims-based active-surveillance alerts as if they were SRS disproportionality — the bias structure and the estimand differ. - Literature / solicited / registry reports: Literature and patient-support-program (solicited) reports follow different reporting dynamics than truly spontaneous ICSRs; mixing them without stratification contaminates the reporting association. Keep report type as a stratification variable.

Worked example (FAERS-style 2×2 for one drug-event pair)

Question: is the pair drug X — rhabdomyolysis a signal of disproportionate reporting in a snapshot of the database? After deduplication, the collapsed contingency table is: `a` = 40 reports naming both drug X and rhabdomyolysis; `b` = 1,960 reports naming drug X with any other event; `c` = 3,000 reports naming rhabdomyolysis with any other drug; `d` = 995,000 all remaining reports. Then: - PRR = [a/(a+b)] / [c/(c+d)] = (40/2000) / (3000/998000) = 0.0200 / 0.003006 = 6.65. - χ² (Yates-corrected, 1 df) on this table ≈ 218 (far exceeds 4). - ROR = (a·d)/(b·c) = (40·995000)/(1960·3000) = 39,800,000 / 5,880,000 = 6.77; 95% CI on the log scale, SE(lnROR) = sqrt(1/a+1/b+1/c+1/d) = sqrt(1/40+1/1960+1/3000+1/995000) = 0.160, so CI = exp(ln6.77 ± 1.96·0.160) = (4.95, 9.27) — lower bound well above 1. - MHRA operational signal criteria: PRR ≥ 2 and χ² ≥ 4 and n (=a) ≥ 3 → 6.65 ≥ 2, 218 ≥ 4, 40 ≥ 3: all met → signal of disproportionate reporting. This is a flag for clinical and epidemiologic review, not a confirmed adverse drug reaction; confirmation requires a denominator-based study.

Interpreting the output

In the worked example, drug X receives a PRR of 6.65 and an ROR of 6.77 (95% CI 4.95–9.27), with all three MHRA criteria satisfied (a = 40, PRR ≥ 2, χ² ≈ 218 ≥ 4).

(1) Formal interpretation. The PRR of 6.65 means that rhabdomyolysis accounts for a 6.65-fold larger share of drug-X reports than it accounts for among all other drugs in the database. The ROR of 6.77 is the odds-ratio analog and is preferred when the event is relatively common in the SRS. Both are statistics derived from a 2×2 table of report counts — there is no exposed or unexposed person-time, no incidence rate, and no background-rate denominator. A 95% CI lower bound well above 1 (here 4.95) addresses within-database sampling uncertainty; it does not address reporting biases such as notoriety bias, stimulated reporting, or differential under-reporting across drugs. The EBGM/IC shrinkage alternatives would pull the estimate closer to the null if a were small, but at a = 40 shrinkage is minimal.

(2) Practical interpretation. A PRR of 6.65 meeting MHRA criteria is a signal of disproportionate reporting, not a confirmed adverse drug reaction and not a risk ratio. The absence of a denominator means you cannot calculate incidence or attribute causation. The appropriate response is clinical and epidemiologic review — for example, a PASS with a proper cohort denominator — before any regulatory or label action. Do not conflate the disproportionality statistic with a rate ratio or an odds ratio from a cohort study; they measure categorically different quantities.

Worked example

Scenario

A pharmacovigilance analyst is screening FDA FAERS to ask whether Drug X is disproportionately reported with rhabdomyolysis (severe muscle breakdown). After deduplication, she collapses the entire database into one 2x2 table. She wants to compute the PRR and ROR and apply the MHRA signal criteria (PRR >= 2 AND chi-square >= 4 AND at least 3 co-reports) to decide whether to flag this pair for review.

Dataset

Collapsed 2x2 contingency table from 1,000,000 deduplicated spontaneous reports. Each cell is a count of reports, not patients.

Steps

• Row totals: reports naming Drug X = a + b = 40 + 1960 = 2000; reports naming rhabdomyolysis with any other drug = c + d = 3000 + 995000 = 998000.

• Drug X reporting proportion for rhabdomyolysis: a / (a+b) = 40 / 2000 = 0.0200 (2.00% of Drug X reports mention rhabdomyolysis).

• Background reporting proportion: c / (c+d) = 3000 / 998000 = 0.003006 (0.30% of all other-drug reports mention rhabdomyolysis).

• PRR = 0.0200 / 0.003006 = 6.65 (rhabdomyolysis is reported about 6.65 times more often with Drug X than with all other drugs).

• ROR = (a x d) / (b x c) = (40 x 995000) / (1960 x 3000) = 39,800,000 / 5,880,000 = 6.77.

• SE of ln(ROR) = sqrt(1/40 + 1/1960 + 1/3000 + 1/995000) = sqrt(0.02500 + 0.00051 + 0.00033 + 0.000001) = sqrt(0.02584) = 0.161.

• 95% CI on ROR = exp(ln(6.77) plus or minus 1.96 x 0.161) = (4.94, 9.28); the lower bound 4.94 is well above 1.

• MHRA signal check: PRR = 6.65 >= 2, chi-square on this table is approximately 218 >= 4, and a = 40 >= 3. All three criteria are met.

Result

PRR = 6.65 and ROR = 6.77 (95% CI 4.94 to 9.28). The MHRA signal criteria are met: Drug X / rhabdomyolysis is a signal of disproportionate reporting and should be forwarded for clinical and epidemiologic review. This means rhabdomyolysis appears in Drug X reports far more often than in the rest of the database, not that Drug X definitely causes rhabdomyolysis.

Runnable example

import pandas as pd
import numpy as np
from scipy.stats import chi2_contingency

def disproportionality(reports: pd.DataFrame,
                       drug_col: str = "drug_name",
                       event_col: str = "event_pt") -> pd.DataFrame:
    # Unique drug-event mentions per report so one ICSR cannot count twice for the same pair.
    pairs = reports[[drug_col, event_col]].drop_duplicates()

    N = reports["report_id"].nunique()                      # total reports = a+b+c+d
    n_drug  = pairs.groupby(drug_col)[event_col].size()     # a+b : reports naming the drug
    n_event = pairs.groupby(event_col)[drug_col].size()     # a+c : reports naming the event
    a_tab   = pairs.groupby([drug_col, event_col]).size()   # a   : reports naming both

    rows = []
    for (drug, event), a in a_tab.items():
        a = int(a)
        b = int(n_drug[drug]  - a)                           # drug, other events
        c = int(n_event[event] - a)                          # event, other drugs
        d = int(N - a - b - c)                               # all other reports
        if min(a, b, c, d) < 0:
            continue

        prr = (a / (a + b)) / (c / (c + d)) if (a + b) and (c + d) else np.nan

        # ROR with log-scale 95% CI; 0.5 continuity correction if any zero cell.
        ca, cb, cc, cd = (a, b, c, d) if min(a, b, c, d) > 0 else (a + .5, b + .5, c + .5, d + .5)
        ror = (ca * cd) / (cb * cc)
        se  = np.sqrt(1/ca + 1/cb + 1/cc + 1/cd)
        ror_lo, ror_hi = np.exp(np.log(ror) - 1.96 * se), np.exp(np.log(ror) + 1.96 * se)

        chi2 = chi2_contingency([[a, b], [c, d]], correction=True)[0]  # Yates-corrected

        expected = (a + b) * (a + c) / N                     # E[a] under independence
        rrr = a / expected if expected else np.nan           # relative reporting ratio (GPS input)

        signal = (prr >= 2) and (chi2 >= 4) and (a >= 3)      # MHRA operational criteria
        rows.append(dict(drug=drug, event=event, a=a, b=b, c=c, d=d,
                         PRR=prr, ROR=ror, ROR_lo=ror_lo, ROR_hi=ror_hi,
                         chi2_yates=chi2, RRR=rrr, MHRA_signal=signal))

    return pd.DataFrame(rows).sort_values("PRR", ascending=False)

library(data.table)

## ---- Canonical route: PhViD (all four estimators incl. Bayesian shrinkage) ----
## library(PhViD)
## dm <- as.PhViD(unique(reports[, c("drug_name", "event_pt", "count")]))  # count optional
## PRR(dm,    RR0 = 1, MIN.n11 = 3)        # PRR with n>=3 gate
## ROR(dm,    RR0 = 1, MIN.n11 = 3)        # reporting odds ratio
## BCPNN(dm,  RR0 = 1, MIN.n11 = 3)        # information component IC025 > 0
## GPS(dm,    RR0 = 1, MIN.n11 = 3)        # EBGM with EB05 >= 2

## ---- Manual fallback (PRR, ROR + 95% CI, Yates chi-square, MHRA flag) ----
disproportionality <- function(reports) {
  dt <- unique(as.data.table(reports)[, .(drug_name, event_pt)])  # one mention per report-pair
  N       <- dt[, uniqueN(.SD)]                                   # a+b+c+d (unique mentions)
  n_drug  <- dt[, .(nd = .N), by = drug_name]
  n_event <- dt[, .(ne = .N), by = event_pt]
  tab     <- dt[, .(a = .N), by = .(drug_name, event_pt)]
  tab <- merge(tab, n_drug,  by = "drug_name")
  tab <- merge(tab, n_event, by = "event_pt")

  tab[, `:=`(b = nd - a, c = ne - a)]
  tab[, d := N - a - b - c]

  tab[, PRR := (a / (a + b)) / (c / (c + d))]
  tab[, `:=`(ca = ifelse(pmin(a,b,c,d) > 0, a, a + .5),
             cb = ifelse(pmin(a,b,c,d) > 0, b, b + .5),
             cc = ifelse(pmin(a,b,c,d) > 0, c, c + .5),
             cd = ifelse(pmin(a,b,c,d) > 0, d, d + .5))]
  tab[, ROR := (ca * cd) / (cb * cc)]
  tab[, se  := sqrt(1/ca + 1/cb + 1/cc + 1/cd)]
  tab[, `:=`(ROR_lo = exp(log(ROR) - 1.96 * se),
             ROR_hi = exp(log(ROR) + 1.96 * se))]

  ## Yates-corrected chi-square on each 2x2.
  tab[, chi2 := {
    E11 <- (a + b) * (a + c) / N
    E12 <- (a + b) * (b + d) / N
    E21 <- (c + d) * (a + c) / N
    E22 <- (c + d) * (b + d) / N
    rowSums(cbind((abs(a - E11) - .5)^2 / E11, (abs(b - E12) - .5)^2 / E12,
                  (abs(c - E21) - .5)^2 / E21, (abs(d - E22) - .5)^2 / E22))
  }, by = .(drug_name, event_pt)]

  tab[, MHRA_signal := PRR >= 2 & chi2 >= 4 & a >= 3]
  tab[order(-PRR), .(drug_name, event_pt, a, b, c, d, PRR, ROR, ROR_lo, ROR_hi, chi2, MHRA_signal)]
}