Safety Signal Case Definition

In plain language

A safety signal case definition is the exact written rule that decides which patients in a health database count as having experienced a specific harmful event, such as serious liver injury. Before researchers can count cases and calculate how often an event occurs, they must specify which medical codes, care settings, and lab results qualify — otherwise two analysts studying the same drug could get completely different counts. The tradeoff at the heart of every case definition is breadth: a broad rule catches most true cases but also mislabels some healthy people as cases (high sensitivity, lower precision), while a narrow rule is more trustworthy per case but may miss real ones (higher precision, lower sensitivity). Getting this choice right determines whether a safety signal is detected at all — or whether it turns out to be noise.

A safety signal case definition is the executable specification that turns a clinical adverse-event concept (e.g., acute liver injury, anaphylaxis, acute kidney injury, rhabdomyolysis) into a reproducible algorithm over claims, EHR, or registry data so that a signal can be evaluated — counted, rate-estimated, and compared across exposures. It is not a single diagnosis code: it is the full rule set — which codes, in which diagnosis position, in which care setting, confirmed by which labs/procedures, within which time window relative to exposure, after excluding which prior history — plus the validation plan (chart-confirmed positive predictive value, sensitivity) that quantifies how much the algorithm misclassifies. In active surveillance systems (FDA Sentinel/ARIA, PRISM, vaccine-safety networks) the case definition is the load-bearing artifact: every rate, sequential test, and signal decision inherits its measurement error.

Core conceptual distinction

— the discriminating feature of a signal-evaluation case definition versus a routine outcome algorithm built for a comparative-effectiveness study is the sensitivity-vs-PPV operating point and the role of downstream confirmation. (1) Signal evaluation favors sensitivity (broad capture) when human/chart adjudication follows: a screening-grade algorithm with modest PPV is acceptable because flagged cases are confirmed before any regulatory action, so the cost of a false positive is a chart review, not a wrong inference. A confirmation-grade definition (high PPV, inpatient-primary-position dx + lab confirmation) is used once a signal must be quantified for a decision. (2) Effectiveness/safety estimation favors PPV-first algorithms because cases are taken at face value into a rate or hazard model and false positives bias the estimate. (3) The estimand the definition feeds matters: an incident (first-ever, new-onset) event with a clean lookback supports an incidence rate or a cause-specific/subdistribution contrast; a prevalent or recurrent capture supports utilization but not incidence. The case definition therefore is not separable from the analysis it serves — it must state incident-vs-prevalent, the etiologically relevant risk window after exposure, and the planned PPV so the downstream rate can be bias-corrected.

Pros, cons, and trade-offs

- vs a single-code, single-position rule (e.g., "any ICD code for the event"): A multi-component definition (dx position + care setting + lab/procedure confirmation + medication corroboration + history exclusion) is far more specific and defensible, raises PPV, and is auditable for regulators. Cost: it requires more programming, code-list curation, and a validation substudy, and an over-specified rule can drop true cases (lower sensitivity), which matters for screening. Prefer the multi-component rule for any consequential signal quantification; a broad single-code rule only as a hypothesis-generating screen with adjudication behind it. - vs a fully chart-adjudicated outcome (gold standard on every case): Adjudication maximizes accuracy but does not scale to a multi-site distributed surveillance network and is impossibly slow for sequential monitoring. A coded case definition + a PPV substudy approximates the truth at population scale. Prefer the coded definition + targeted chart-review PPV for surveillance; reserve full adjudication for confirmation of a strong signal. - vs reusing a published validated algorithm verbatim: Borrowing a validated definition (e.g., a Mini-Sentinel/Sentinel algorithm) saves work and imports a PPV estimate. Cost: PPV is not transportable — it depends on outcome prevalence, coding era (ICD-9 vs ICD-10), data source, and care setting, so an imported PPV can be wrong in your population. Prefer borrowing the code-list logic but re-estimating PPV locally whenever the source population differs.

When to use

— active or scheduled safety surveillance and signal evaluation in claims/EHR/registry data (Sentinel/ARIA, PRISM, vaccine networks, manufacturer post-marketing commitments); any RWE study whose endpoint is a serious adverse event where coded data alone misclassify materially; settings that need a transparent, pre-registered, validation-backed outcome rule to satisfy FDA/EMA expectations for fit-for-purpose data and reliability. Use it whenever the count of events drives a regulatory or labeling decision and the algorithm's error must be quantified, not assumed away.

When NOT to use — and when it is actively misleading or dangerous

- No validation/PPV is or will be available. Reporting an adverse-event rate from an unvalidated algorithm as if it were the true rate is the central trap of signal evaluation: differential misclassification by exposure (e.g., a drug that triggers more lab testing, hence more incidentally coded events) creates a spurious signal or masks a real one. Without a PPV (and ideally sensitivity), the rate is uninterpretable. - The event is non-specifically coded. Outcomes whose codes are dominated by rule-out/symptom coding (e.g., chest pain for MI, transaminase elevation for "liver injury") have low PPV that no clever logic fully rescues; a broad definition here manufactures cases. Require lab/procedure confirmation or do not quantify. - Recurrent/prevalent capture passed off as incidence. Counting a chronic or recurrent condition as if each code were a new event inflates rates and breaks the incidence estimand; demand an event-free lookback and a recurrence rule. - Imported PPV across a coding transition or a different data source. Applying an ICD-9-era PPV to ICD-10 data, or a commercial-claims PPV to a Medicare FFS elderly population, silently rebiases the corrected rate. - Immortal-time or look-ahead leakage in the definition. If the case window or a confirmation requirement uses information that accrues only to survivors (e.g., requiring a 30-day follow-up lab), the rule conditions on the future and biases the comparison.

Data-source operational depth

- Claims (FFS vs Medicare Advantage): The event is a constellation of medical claims — inpatient principal vs secondary diagnosis position, place-of-service (ER vs inpatient vs office), procedure/revenue codes, and corroborating drug dispensings. The dominant failure mode is incomplete capture in Medicare Advantage and capitated/bundled arrangements: encounter data are inconsistently submitted, so MA-only person-time can miss events entirely — restrict to enrollees with complete medical (Parts A/B) capture or treat MA spans as unobserved, never as event-free. Claims adjudication lag and reversals mean recent person-time is artificially event-sparse; impose a data-maturity buffer. Diagnosis position drives PPV (a principal inpatient code is far more specific than a same-day ER rule-out code). Differential competing risks by exposure in the elderly (a drug used in sicker patients sees more death before the event) bias a naive cumulative-incidence comparison — pre-specify cause-specific vs subdistribution. - EHR: Labs, vitals, problem lists, and orders enable confirmation-grade definitions (e.g., ALT/AST thresholds for liver injury, troponin for MI) that claims cannot match — the major advantage. But capture is encounter-driven and leaky: care outside the system is invisible, so a true event treated at an outside ED is missed, and structured fields are often blank with the signal buried in notes (requires NLP). Site workflow heterogeneity makes a fixed lab threshold behave differently across sites. - Registry: Best for adjudicated, clinically rich outcomes (e.g., cancer, MI registries) but typically incomplete for full longitudinal capture and weak on exposure; check registration completeness and adjudication rules, and link to claims for denominator person-time. - Linked claims–EHR–vital records: The ideal substrate — EHR labs to confirm + claims completeness + a death index to handle the competing risk — but linkage selection (only the linkable subset) and order/fill/service-date discrepancies must be reconciled before the case window is anchored, or the algorithm dates events incorrectly.

Worked claims example (acute liver injury signal, commercial + Medicare FFS)

Goal: a confirmation-grade incident-ALI algorithm to quantify a hepatotoxicity signal for a newly approved drug. (1) Continuous enrollment / observability: require continuous medical + pharmacy enrollment with complete FFS capture for the 183-day lookback through follow-up; drop MA-only person-time. (2) Case logic: an inpatient claim with ALI in the principal diagnosis position (`dx_position = 1`, `pos = '21'`) OR an ER claim with an ALI diagnosis plus a same-episode liver-function-test claim (corroboration), within the etiologically relevant risk window of 1–90 days after the index dispensing (`days_supply`-anchored on-treatment time + grace). (3) Incident-event coding: exclude anyone with any ALI diagnosis or chronic-liver-disease code in the 183-day lookback so the captured event is first-ever; take the first qualifying claim date as the event date. (4) Validation: a random sample of 200 algorithm-positive patients undergoes chart review; 162 confirm → PPV = 0.81 (95% Wilson CI 0.75–0.86). (5) Bias-corrected rate: if the crude algorithm-based rate is 12.0 / 1,000 person-years, the PPV-corrected true-positive rate is 12.0 × 0.81 ≈ 9.7 / 1,000 PY (assuming negligible false-negative impact on the numerator; a full correction also uses sensitivity — see misclassification-bias-correction-rwe). (6) Signal logic: a screening run might instead drop the lab confirmation to raise sensitivity, accepting PPV ≈ 0.55, because every flagged case is adjudicated before the signal is acted on — the same clinical concept, a different operating point chosen deliberately for the surveillance stage.

Interpreting the output

. The confirmatory ALI case definition (inpatient principal position plus an ALT/AST elevation lab confirmatory requirement) produces a chart-review-validated PPV = 0.81 (95% Wilson CI 0.75–0.86) from 200 randomly sampled algorithm-positive patients, of whom 162 were confirmed as true ALI cases. Applying the PPV correction to the crude algorithm-based incidence rate of 12.0 per 1,000 person-years gives a corrected rate of approximately 9.7 per 1,000 person-years.

Formal interpretation: PPV = 0.81 means 19 of every 100 algorithm-positive patients are false positives — patients who met the code-and-lab criteria but whose charts did not confirm ALI to clinical adjudication standards. The crude rate of 12.0 per 1,000 PY therefore overstates the true-positive rate by roughly 19%; the corrected rate of 9.7 per 1,000 PY is the preferred epidemiologic estimate for regulatory and HTA audiences. The PPV alone does not characterize sensitivity — the algorithm may still miss a substantial fraction of true ALI cases that were never coded or tested, and those missed cases would require a separately designed sensitivity estimation (gold-standard-linked substudy).

Practical interpretation: the choice of operating point is a pre-specified analytical decision, not a post-hoc optimization. A screening-stage run that relaxes the lab confirmation requirement raises sensitivity at the cost of PPV (here approximately 0.55), which is appropriate when every flagged case will be manually adjudicated before any regulatory action is taken. Document both operating points in the pharmacovigilance protocol, alongside the chart-review protocol and adjudicator credentials.

Worked example

Scenario

A team is studying whether a new cholesterol-lowering drug causes acute liver injury (ALI). They need to define what counts as an ALI case in a commercial insurance claims database before they can count events and compare rates between users and non-users. They draft two candidate definitions — a broad one and a narrow one — and then reason through which patients each definition captures and misses.

Dataset

Five candidate patients and the evidence available for each. The team must decide whether each counts as an ALI case under a broad vs. narrow definition.

Steps

• Define the broad rule: any ALI diagnosis code (K71.6) appearing anywhere on any claim in the risk window counts as a case, regardless of where on the form it appears or where care was received.

• Under the broad rule, all five patients (P-101 through P-105) are flagged as algorithm-positive cases.

• Now assess which of these are likely true ALI: P-101 is almost certainly real (hospital admission, principal diagnosis). P-102 is plausible (ER visit with a supporting lab test). P-103 is doubtful (ER rule-out with no lab confirmation). P-104 is very unlikely to be true ALI (incidental office code, no workup). P-105 is ambiguous (secondary complication during an unrelated admission).

• Define the narrow rule: ALI must appear as the principal (first-listed) diagnosis on an inpatient claim, OR as any position on an ER claim AND have a liver lab test on the same date.

• Under the narrow rule, only P-101 (inpatient, principal) and P-102 (ER plus lab) are flagged. P-103, P-104, and P-105 are excluded.

• The tradeoff: if 10 patients truly had drug-induced ALI during follow-up and 8 of them were hospitalized or had a confirmed ER visit (captured by the narrow rule), while 2 were only captured by the office code (captured only by the broad rule), then: broad rule sensitivity = 10/10 = 1.00 but PPV = 10/37 = 0.27 if 27 false-positive office/ER rule-out codes also exist; narrow rule sensitivity = 8/10 = 0.80 but PPV = 8/9 = 0.89 if only 1 of the 9 flagged narrow-rule cases is a false positive.

• The team records the sensitivity and PPV for each definition and chooses based on the study goal: early screening (favor broad, use chart review to confirm) vs. final rate estimation (favor narrow, accept missing a few real cases to keep the count trustworthy).

Result

Narrow rule: flags P-101 and P-102 only. Estimated PPV 0.89 (8 true cases out of 9 flagged), estimated sensitivity 0.80 (captures 8 of 10 true cases). Broad rule: flags all five patients. Estimated PPV 0.27 (10 true out of 37 broad-rule positives in the full cohort), estimated sensitivity 1.00 (catches all 10 true cases). Implication: the narrow definition produces a more trustworthy case count for rate estimation; the broad definition is appropriate only if every flagged patient will be individually reviewed before any conclusion is drawn.

Runnable example

import pandas as pd
import numpy as np
from scipy import stats

LOOKBACK_DAYS = 183       # event-free window that makes the captured event incident
RISK_START, RISK_END = 1, 90   # etiologically relevant post-index risk window (days)
ALI_DX = {"K71", "K72", "B190", "K762", "K729"}  # ALI / hepatic-failure code prefixes (illustrative)
CHRONIC_LIVER = {"K70", "K74", "B18", "K766"}    # chronic-liver-disease prefixes to exclude (prevalent)

def _has_prefix(s, prefixes):
    return s.str.startswith(tuple(prefixes))

def build_ali_cases(dx, lab, cohort):
    dx = dx.merge(cohort[["person_id", "index_date", "fup_end"]], on="person_id")

    # 1) Exclude prevalent disease: any ALI or chronic-liver code in the event-free lookback.
    look = dx[(dx["from_dt"] < dx["index_date"]) &
              (dx["from_dt"] >= dx["index_date"] - pd.Timedelta(days=LOOKBACK_DAYS)) &
              (_has_prefix(dx["icd_dx"], ALI_DX | CHRONIC_LIVER))]
    incident_ids = set(cohort["person_id"]) - set(look["person_id"])

    # 2) In-window ALI claims: principal inpatient dx OR ER ALI dx corroborated by a same-day LFT claim.
    w = dx[dx["person_id"].isin(incident_ids)].copy()
    w["day"] = (w["from_dt"] - w["index_date"]).dt.days
    w = w[(w["day"] >= RISK_START) & (w["day"] <= RISK_END) &
          (w["from_dt"] <= w["fup_end"]) & _has_prefix(w["icd_dx"], ALI_DX)]

    inpatient_principal = w[(w["dx_position"] == 1) & (w["pos"] == "21")]
    er = w[w["pos"] == "23"]
    er_corrob = er.merge(lab[lab["lab_flag"] == 1][["person_id", "from_dt"]],
                         on=["person_id", "from_dt"], how="inner")
    qualifying = pd.concat([inpatient_principal, er_corrob], ignore_index=True)

    # 3) First-event coding: earliest qualifying claim is the incident event date.
    cases = (qualifying.sort_values("from_dt")
                       .groupby("person_id", as_index=False)
                       .first()[["person_id", "from_dt"]]
                       .rename(columns={"from_dt": "event_date"}))
    return cases

def crude_rate(cases, cohort, per=1000):
    py = ((cohort["fup_end"] - cohort["index_date"]).dt.days.clip(lower=0).sum()) / 365.25
    return per * len(cases) / py, py

def wilson_ci(k, n, alpha=0.05):
    z = stats.norm.ppf(1 - alpha / 2)
    p = k / n
    denom = 1 + z**2 / n
    center = (p + z**2 / (2 * n)) / denom
    half = z * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom
    return p, center - half, center + half

cases = build_ali_cases(dx, lab, cohort)
rate, py = crude_rate(cases, cohort)
ppv, lo, hi = wilson_ci(int(review["confirmed"].sum()), len(review))
corrected_rate = rate * ppv    # numerator-only correction; full correction also needs sensitivity
print(f"crude={rate:.2f}/1000PY  PPV={ppv:.2f} (95% CI {lo:.2f}-{hi:.2f})  corrected={corrected_rate:.2f}/1000PY")

library(data.table)
LOOKBACK_DAYS <- 183L
RISK_START <- 1L; RISK_END <- 90L
ALI_DX        <- c("K71","K72","B190","K762","K729")
CHRONIC_LIVER <- c("K70","K74","B18","K766")
has_prefix <- function(x, pref) Reduce(`|`, lapply(pref, function(p) startsWith(x, p)))

build_ali_cases <- function(dx, lab, cohort) {
  setDT(dx); setDT(lab); setDT(cohort)
  dx <- merge(dx, cohort[, .(person_id, index_date, fup_end)], by = "person_id")

  # 1) Exclude prevalent disease in the event-free lookback.
  look <- dx[from_dt < index_date & from_dt >= index_date - LOOKBACK_DAYS &
             has_prefix(icd_dx, c(ALI_DX, CHRONIC_LIVER)), unique(person_id)]
  incident_ids <- setdiff(cohort$person_id, look)

  # 2) In-window ALI: principal inpatient dx OR ER ALI dx corroborated by a same-day LFT claim.
  w <- dx[person_id %in% incident_ids]
  w[, day := as.integer(from_dt - index_date)]
  w <- w[day >= RISK_START & day <= RISK_END & from_dt <= fup_end & has_prefix(icd_dx, ALI_DX)]

  inpatient_principal <- w[dx_position == 1L & pos == "21"]
  er <- w[pos == "23"]
  lftd <- lab[lab_flag == 1L, .(person_id, from_dt)]
  er_corrob <- merge(er, lftd, by = c("person_id", "from_dt"))
  qualifying <- rbindlist(list(inpatient_principal, er_corrob), use.names = TRUE, fill = TRUE)

  # 3) First-event coding.
  setorder(qualifying, from_dt)
  qualifying[, .(event_date = from_dt[1L]), by = person_id]
}

wilson_ci <- function(k, n, alpha = 0.05) {
  z <- qnorm(1 - alpha / 2); p <- k / n; denom <- 1 + z^2 / n
  center <- (p + z^2 / (2 * n)) / denom
  half <- z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2)) / denom
  c(ppv = p, lo = center - half, hi = center + half)
}

cases <- build_ali_cases(dx, lab, cohort)
setDT(cohort)
py <- sum(pmax(as.integer(cohort$fup_end - cohort$index_date), 0)) / 365.25
rate <- 1000 * nrow(cases) / py
w <- wilson_ci(sum(review$confirmed), nrow(review))
cat(sprintf("crude=%.2f/1000PY  PPV=%.2f (95%% CI %.2f-%.2f)  corrected=%.2f/1000PY\n",
            rate, w["ppv"], w["lo"], w["hi"], rate * w["ppv"]))