← Methods repository
REPOSITORY / OUTCOME MEASURE / SAFETY SIGNAL CASE DEFINITION
CONCEPTINTERMEDIATEPYTHON · R · SAS4 citations

Safety Signal Case Definition

An operational, pre-specified rule that identifies suspected adverse-event cases in routinely-collected data for active safety surveillance, combining diagnosis, procedure, laboratory, medication, timing, and care-setting evidence with a stated incident-event window and a validation/PPV plan.

Outcome Measuresafety-signalcase-definitionadverse-event-algorithmpharmacovigilanceactive-surveillancepositive-predictive-valueoutcome-validation
On this page
Methods reference only. Use primary source citations and local policy before applying this in a study protocol, regulatory submission, payer dossier, or clinical decision.
In plain language

A safety signal case definition is the exact written rule that decides which patients in a health database count as having experienced a specific harmful event, such as serious liver injury. Before researchers can count cases and calculate how often an event occurs, they must specify which medical codes, care settings, and lab results qualify — otherwise two analysts studying the same drug could get completely different counts. The tradeoff at the heart of every case definition is breadth: a broad rule catches most true cases but also mislabels some healthy people as cases (high sensitivity, lower precision), while a narrow rule is more trustworthy per case but may miss real ones (higher precision, lower sensitivity). Getting this choice right determines whether a safety signal is detected at all — or whether it turns out to be noise.

When to use it
—Any signal quantification feeding a regulatory or labeling decision; reserve single-code rules for hypothesis-generating screens backed by adjudication.
—Routine and sequential surveillance; reserve full adjudication for confirming a strong, decision-relevant signal.
—Borrow the code-list logic but re-estimate PPV locally whenever the source population or coding era differs from the original validation.
Watch out for
—More programming and code-list curation, requires a validation substudy, and over-specification can lower sensitivity (drop true cases) for screening.
—Coded capture is imperfect (PPV<1, sensitivity<1) and inherits all administrative-data measurement error.
—PPV is not transportable across outcome prevalence, coding era (ICD-9/ICD-10), data source, and care setting; the imported PPV can rebias the corrected rate.

A safety signal case definition is the executable specification that turns a clinical adverse-event concept (e.g., acute liver injury, anaphylaxis, acute kidney injury, rhabdomyolysis) into a reproducible algorithm over claims, EHR, or registry data so that a signal can be evaluated — counted, rate-estimated, and compared across exposures. It is not a single diagnosis code: it is the full rule set — which codes, in which diagnosis position, in which care setting, confirmed by which labs/procedures, within which time window relative to exposure, after excluding which prior history — plus the validation plan (chart-confirmed positive predictive value, sensitivity) that quantifies how much the algorithm misclassifies. In active surveillance systems (FDA Sentinel/ARIA, PRISM, vaccine-safety networks) the case definition is the load-bearing artifact: every rate, sequential test, and signal decision inherits its measurement error.

Core conceptual distinction

the discriminating feature of a signal-evaluation case definition versus a routine outcome algorithm built for a comparative-effectiveness study is the sensitivity-vs-PPV operating point and the role of downstream confirmation.

  1. Signal evaluation favors sensitivity (broad capture) when human/chart adjudication follows: a screening-grade algorithm with modest PPV is acceptable because flagged cases are confirmed before any regulatory action, so the cost of a false positive is a chart review, not a wrong inference. A confirmation-grade definition (high PPV, inpatient-primary-position dx + lab confirmation) is used once a signal must be quantified for a decision.
  2. Effectiveness/safety estimation favors PPV-first algorithms because cases are taken at face value into a rate or hazard model and false positives bias the estimate.
  3. The estimand the definition feeds matters: an incident (first-ever, new-onset) event with a clean lookback supports an incidence rate or a cause-specific/subdistribution contrast; a prevalent or recurrent capture supports utilization but not incidence. The case definition therefore is not separable from the analysis it serves — it must state incident-vs-prevalent, the etiologically relevant risk window after exposure, and the planned PPV so the downstream rate can be bias-corrected.

Pros, cons, and trade-offs

  • vs a single-code, single-position rule (e.g., "any ICD code for the event"): A multi-component definition (dx position + care setting + lab/procedure confirmation + medication corroboration + history exclusion) is far more specific and defensible, raises PPV, and is auditable for regulators. Cost: it requires more programming, code-list curation, and a validation substudy, and an over-specified rule can drop true cases (lower sensitivity), which matters for screening. Prefer the multi-component rule for any consequential signal quantification; a broad single-code rule only as a hypothesis-generating screen with adjudication behind it.
  • vs a fully chart-adjudicated outcome (gold standard on every case): Adjudication maximizes accuracy but does not scale to a multi-site distributed surveillance network and is impossibly slow for sequential monitoring. A coded case definition + a PPV substudy approximates the truth at population scale. Prefer the coded definition + targeted chart-review PPV for surveillance; reserve full adjudication for confirmation of a strong signal.
  • vs reusing a published validated algorithm verbatim: Borrowing a validated definition (e.g., a Mini-Sentinel/Sentinel algorithm) saves work and imports a PPV estimate. Cost: PPV is not transportable — it depends on outcome prevalence, coding era (ICD-9 vs ICD-10), data source, and care setting, so an imported PPV can be wrong in your population. Prefer borrowing the code-list logic but re-estimating PPV locally whenever the source population differs.

When NOT to use — and when it is actively misleading or dangerous

  • No validation/PPV is or will be available. Reporting an adverse-event rate from an unvalidated algorithm as if it were the true rate is the central trap of signal evaluation: differential misclassification by exposure (e.g., a drug that triggers more lab testing, hence more incidentally coded events) creates a spurious signal or masks a real one. Without a PPV (and ideally sensitivity), the rate is uninterpretable.
  • The event is non-specifically coded. Outcomes whose codes are dominated by rule-out/symptom coding (e.g., chest pain for MI, transaminase elevation for "liver injury") have low PPV that no clever logic fully rescues; a broad definition here manufactures cases. Require lab/procedure confirmation or do not quantify.
  • Recurrent/prevalent capture passed off as incidence. Counting a chronic or recurrent condition as if each code were a new event inflates rates and breaks the incidence estimand; demand an event-free lookback and a recurrence rule.
  • Imported PPV across a coding transition or a different data source. Applying an ICD-9-era PPV to ICD-10 data, or a commercial-claims PPV to a Medicare FFS elderly population, silently rebiases the corrected rate.
  • Immortal-time or look-ahead leakage in the definition. If the case window or a confirmation requirement uses information that accrues only to survivors (e.g., requiring a 30-day follow-up lab), the rule conditions on the future and biases the comparison.

Data-source operational depth

  • Claims (FFS vs Medicare Advantage): The event is a constellation of medical claims — inpatient principal vs secondary diagnosis position, place-of-service (ER vs inpatient vs office), procedure/revenue codes, and corroborating drug dispensings. The dominant failure mode is incomplete capture in Medicare Advantage and capitated/bundled arrangements: encounter data are inconsistently submitted, so MA-only person-time can miss events entirely — restrict to enrollees with complete medical (Parts A/B) capture or treat MA spans as unobserved, never as event-free. Claims adjudication lag and reversals mean recent person-time is artificially event-sparse; impose a data-maturity buffer. Diagnosis position drives PPV (a principal inpatient code is far more specific than a same-day ER rule-out code). Differential competing risks by exposure in the elderly (a drug used in sicker patients sees more death before the event) bias a naive cumulative-incidence comparison — pre-specify cause-specific vs subdistribution.
  • EHR: Labs, vitals, problem lists, and orders enable confirmation-grade definitions (e.g., ALT/AST thresholds for liver injury, troponin for MI) that claims cannot match — the major advantage. But capture is encounter-driven and leaky: care outside the system is invisible, so a true event treated at an outside ED is missed, and structured fields are often blank with the signal buried in notes (requires NLP). Site workflow heterogeneity makes a fixed lab threshold behave differently across sites.
  • Registry: Best for adjudicated, clinically rich outcomes (e.g., cancer, MI registries) but typically incomplete for full longitudinal capture and weak on exposure; check registration completeness and adjudication rules, and link to claims for denominator person-time.
  • Linked claims–EHR–vital records: The ideal substrate — EHR labs to confirm + claims completeness + a death index to handle the competing risk — but linkage selection (only the linkable subset) and order/fill/service-date discrepancies must be reconciled before the case window is anchored, or the algorithm dates events incorrectly.

Worked claims example (acute liver injury signal, commercial + Medicare FFS)

Goal: a confirmation-grade incident-ALI algorithm to quantify a hepatotoxicity signal for a newly approved drug.

  1. Continuous enrollment / observability: require continuous medical + pharmacy enrollment with complete FFS capture for the 183-day lookback through follow-up; drop MA-only person-time.
  2. Case logic: an inpatient claim with ALI in the principal diagnosis position (`dx_position = 1`, `pos = '21'`) OR an ER claim with an ALI diagnosis plus a same-episode liver-function-test claim (corroboration), within the etiologically relevant risk window of 1–90 days after the index dispensing (`days_supply`-anchored on-treatment time + grace).
  3. Incident-event coding: exclude anyone with any ALI diagnosis or chronic-liver-disease code in the 183-day lookback so the captured event is first-ever; take the first qualifying claim date as the event date.
  4. Validation: a random sample of 200 algorithm-positive patients undergoes chart review; 162 confirm → PPV = 0.81 (95% Wilson CI 0.75–0.86).
  5. Bias-corrected rate: if the crude algorithm-based rate is 12.0 / 1,000 person-years, the PPV-corrected true-positive rate is 12.0 × 0.81 ≈ 9.7 / 1,000 PY (assuming negligible false-negative impact on the numerator; a full correction also uses sensitivity — see misclassification-bias-correction-rwe).
  6. Signal logic: a screening run might instead drop the lab confirmation to raise sensitivity, accepting PPV ≈ 0.55, because every flagged case is adjudicated before the signal is acted on — the same clinical concept, a different operating point chosen deliberately for the surveillance stage.

Interpreting the output

. The confirmatory ALI case definition (inpatient principal position plus an ALT/AST elevation lab confirmatory requirement) produces a chart-review-validated PPV = 0.81 (95% Wilson CI 0.75–0.86) from 200 randomly sampled algorithm-positive patients, of whom 162 were confirmed as true ALI cases. Applying the PPV correction to the crude algorithm-based incidence rate of 12.0 per 1,000 person-years gives a corrected rate of approximately 9.7 per 1,000 person-years.

Formal interpretation: PPV = 0.81 means 19 of every 100 algorithm-positive patients are false positives — patients who met the code-and-lab criteria but whose charts did not confirm ALI to clinical adjudication standards. The crude rate of 12.0 per 1,000 PY therefore overstates the true-positive rate by roughly 19%; the corrected rate of 9.7 per 1,000 PY is the preferred epidemiologic estimate for regulatory and HTA audiences. The PPV alone does not characterize sensitivity — the algorithm may still miss a substantial fraction of true ALI cases that were never coded or tested, and those missed cases would require a separately designed sensitivity estimation (gold-standard-linked substudy).

Practical interpretation: the choice of operating point is a pre-specified analytical decision, not a post-hoc optimization. A screening-stage run that relaxes the lab confirmation requirement raises sensitivity at the cost of PPV (here approximately 0.55), which is appropriate when every flagged case will be manually adjudicated before any regulatory action is taken. Document both operating points in the pharmacovigilance protocol, alongside the chart-review protocol and adjudicator credentials.

Decision diagram

flowchart TD
  Src[Exposed cohort with complete FFS medical capture<br/>continuous enrollment across lookback + follow-up] --> Look{ALI or chronic-liver code<br/>in 183-day event-free lookback?}
  Look -->|Yes| Excl[Exclude: prevalent / recurrent disease]
  Look -->|No| Win[In risk window 1-90d post-index?]
  Win -->|No| Drop[Not a case in this window]
  Win -->|Yes| Logic{Case logic}
  Logic -->|Inpatient principal dx<br/>dx_position=1, pos=21| Case[Incident case = first qualifying claim date]
  Logic -->|ER ALI dx + same-day LFT claim| Case
  Case --> PPV[Chart-review PPV on a random positive sample]
  PPV --> Corr[PPV-corrected rate = crude rate x PPV<br/>full correction also uses sensitivity]
  Corr --> Sig[Signal evaluation: sequential test / rate comparison]
Decision logic for a confirmation-grade incident acute-liver-injury case definition, from observability and the event-free lookback through the dual case rule, first-event coding, chart-review PPV, and PPV-corrected rate that feeds signal evaluation.
flowchart LR
  A[Screening stage:<br/>high-sensitivity broad rule<br/>any-position dx, no lab] --> B[Flag candidate cases]
  B --> C[Chart adjudication of every flagged case]
  C --> D{Signal confirmed?}
  D -->|No| E[Close: false-positive coding]
  D -->|Yes| F[Confirmation stage:<br/>high-PPV rule<br/>inpatient principal dx + lab]
  F --> G[Quantify: PPV-corrected rate + sequential test]
  G --> H[Regulatory / labeling decision]
The signal-evaluation operating-point shift that distinguishes this concept from a generic outcome algorithm: a high-sensitivity screen with mandatory adjudication generates hypotheses, then a high-PPV confirmation definition quantifies the signal for a decision.

Worked example

Scenario

A team is studying whether a new cholesterol-lowering drug causes acute liver injury (ALI). They need to define what counts as an ALI case in a commercial insurance claims database before they can count events and compare rates between users and non-users. They draft two candidate definitions — a broad one and a narrow one — and then reason through which patients each definition captures and misses.

Dataset

Five candidate patients and the evidence available for each. The team must decide whether each counts as an ALI case under a broad vs. narrow definition.

patient_idevent_descriptiondx_codedx_positioncare_settingliver_lab_flag
P-101Admitted to hospital, ALI listed as the main reasonK71.6principal (1st)inpatientyes
P-102ER visit, ALI code listed, liver enzyme test ordered same dayK71.6secondary (2nd)emergency roomyes
P-103ER visit, ALI code listed as a rule-out, no lab orderedK71.6secondary (2nd)emergency roomno
P-104Routine office visit, ALI code appears incidentallyK71.6secondary (2nd)outpatient officeno
P-105Admitted to hospital, ALI listed as a secondary complicationK71.6secondary (2nd)inpatientno

Steps

1Define the broad rule: any ALI diagnosis code (K71.6) appearing anywhere on any claim in the risk window counts as a case, regardless of where on the form it appears or where care was received.
2Under the broad rule, all five patients (P-101 through P-105) are flagged as algorithm-positive cases.
3Now assess which of these are likely true ALI: P-101 is almost certainly real (hospital admission, principal diagnosis). P-102 is plausible (ER visit with a supporting lab test). P-103 is doubtful (ER rule-out with no lab confirmation). P-104 is very unlikely to be true ALI (incidental office code, no workup). P-105 is ambiguous (secondary complication during an unrelated admission).
4Define the narrow rule: ALI must appear as the principal (first-listed) diagnosis on an inpatient claim, OR as any position on an ER claim AND have a liver lab test on the same date.
5Under the narrow rule, only P-101 (inpatient, principal) and P-102 (ER plus lab) are flagged. P-103, P-104, and P-105 are excluded.
6The tradeoff: if 10 patients truly had drug-induced ALI during follow-up and 8 of them were hospitalized or had a confirmed ER visit (captured by the narrow rule), while 2 were only captured by the office code (captured only by the broad rule), then: broad rule sensitivity = 10/10 = 1.00 but PPV = 10/37 = 0.27 if 27 false-positive office/ER rule-out codes also exist; narrow rule sensitivity = 8/10 = 0.80 but PPV = 8/9 = 0.89 if only 1 of the 9 flagged narrow-rule cases is a false positive.
7The team records the sensitivity and PPV for each definition and chooses based on the study goal: early screening (favor broad, use chart review to confirm) vs. final rate estimation (favor narrow, accept missing a few real cases to keep the count trustworthy).

Result

Narrow rule: flags P-101 and P-102 only. Estimated PPV 0.89 (8 true cases out of 9 flagged), estimated sensitivity 0.80 (captures 8 of 10 true cases). Broad rule: flags all five patients. Estimated PPV 0.27 (10 true out of 37 broad-rule positives in the full cohort), estimated sensitivity 1.00 (catches all 10 true cases). Implication: the narrow definition produces a more trustworthy case count for rate estimation; the broad definition is appropriate only if every flagged patient will be individually reviewed before any conclusion is drawn.

Trade-offs

vs. Single code, single position rule
Pros of this
—Multi-component logic (dx position + care setting + lab/procedure confirmation + history exclusion) raises PPV, is auditable for regulators, and ties the captured event to an explicit incident estimand.
vs. Full chart adjudication of every case
Pros of this
—Scales to multi-site distributed networks and sequential monitoring; a coded definition plus a PPV substudy approximates the truth at population scale.
vs. Reusing a published validated algorithm verbatim (imported PPV)
Pros of this
—Saves curation effort and imports a documented code list and operating characteristics.

Runnable example

Confirmation-grade incident acute-liver-injury (ALI) case definition from claims, with chart-review PPV and a PPV-corrected rate. Required inputs (cleaned, de-duplicated): dx : medical-claim diagnoses -> person_id, clm_id, from_dt (datetime), icd_dx, dx_position (1=principal), pos (place-of-service) lab :...

requires: pandas · numpy · scipy
import pandas as pd
import numpy as np
from scipy import stats

LOOKBACK_DAYS = 183       # event-free window that makes the captured event incident
RISK_START, RISK_END = 1, 90   # etiologically relevant post-index risk window (days)
ALI_DX = {"K71", "K72", "B190", "K762", "K729"}  # ALI / hepatic-failure code prefixes (illustrative)
CHRONIC_LIVER = {"K70", "K74", "B18", "K766"}    # chronic-liver-disease prefixes to exclude (prevalent)

def _has_prefix(s, prefixes):
    return s.str.startswith(tuple(prefixes))

def build_ali_cases(dx, lab, cohort):
    dx = dx.merge(cohort[["person_id", "index_date", "fup_end"]], on="person_id")

    # 1) Exclude prevalent disease: any ALI or chronic-liver code in the event-free lookback.
    look = dx[(dx["from_dt"] < dx["index_date"]) &
              (dx["from_dt"] >= dx["index_date"] - pd.Timedelta(days=LOOKBACK_DAYS)) &
              (_has_prefix(dx["icd_dx"], ALI_DX | CHRONIC_LIVER))]
    incident_ids = set(cohort["person_id"]) - set(look["person_id"])

    # 2) In-window ALI claims: principal inpatient dx OR ER ALI dx corroborated by a same-day LFT claim.
    w = dx[dx["person_id"].isin(incident_ids)].copy()
    w["day"] = (w["from_dt"] - w["index_date"]).dt.days
    w = w[(w["day"] >= RISK_START) & (w["day"] <= RISK_END) &
          (w["from_dt"] <= w["fup_end"]) & _has_prefix(w["icd_dx"], ALI_DX)]

    inpatient_principal = w[(w["dx_position"] == 1) & (w["pos"] == "21")]
    er = w[w["pos"] == "23"]
    er_corrob = er.merge(lab[lab["lab_flag"] == 1][["person_id", "from_dt"]],
                         on=["person_id", "from_dt"], how="inner")
    qualifying = pd.concat([inpatient_principal, er_corrob], ignore_index=True)

    # 3) First-event coding: earliest qualifying claim is the incident event date.
    cases = (qualifying.sort_values("from_dt")
                       .groupby("person_id", as_index=False)
                       .first()[["person_id", "from_dt"]]
                       .rename(columns={"from_dt": "event_date"}))
    return cases

def crude_rate(cases, cohort, per=1000):
    py = ((cohort["fup_end"] - cohort["index_date"]).dt.days.clip(lower=0).sum()) / 365.25
    return per * len(cases) / py, py

def wilson_ci(k, n, alpha=0.05):
    z = stats.norm.ppf(1 - alpha / 2)
    p = k / n
    denom = 1 + z**2 / n
    center = (p + z**2 / (2 * n)) / denom
    half = z * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom
    return p, center - half, center + half

cases = build_ali_cases(dx, lab, cohort)
rate, py = crude_rate(cases, cohort)
ppv, lo, hi = wilson_ci(int(review["confirmed"].sum()), len(review))
corrected_rate = rate * ppv    # numerator-only correction; full correction also needs sensitivity
print(f"crude={rate:.2f}/1000PY  PPV={ppv:.2f} (95% CI {lo:.2f}-{hi:.2f})  corrected={corrected_rate:.2f}/1000PY")

Citations

FOUNDATIONAL / METHODS
  1. [1]Benchimol EI, Smeeth L, Guttmann A, et al. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) Statement. PLoS Medicine. 2015;12(10):e1001885.
  2. [2]Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527.
APPLIED EXAMPLES
  1. [3]Andrade SE, Harrold LR, Tjia J, et al. A systematic review of validated methods for identifying cerebrovascular accident or transient ischemic attack using administrative data. Pharmacoepidemiology and Drug Safety. 2012;21(S1):100-128.
REPORTING & GUIDANCE
  1. [4]Berger ML, Sox H, Willke RJ, et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making. Pharmacoepidemiology and Drug Safety. 2017;26(9):1033-1039.