Risk Evaluation Study (Post-Authorization Safety / Active Surveillance)

In plain language

A risk evaluation study answers the question: how often does a specific harmful side effect actually happen in patients who take a drug in everyday clinical care, and — when a formal safety program is in place — is that program keeping the harm under control? Researchers identify every new patient who started the drug, track how long each person was on it, and count who experienced the safety problem; dividing those counts gives an incidence rate the FDA or EMA can use to update labeling or judge whether a safety program is working. One important caveat: this approach needs a large database of real insurance claims or medical records — it cannot work from reports that patients or doctors voluntarily phone in, because those reports have no reliable count of the people who did NOT have the problem.

A risk evaluation study is a post-authorization safety study (PASS / active surveillance) whose purpose is to measure a drug's risk in routine use — the absolute incidence of a pre-specified adverse event (AE), how that risk compares to an appropriate reference, and, when the question is regulatory, whether risk-minimization measures are achieving their stated objectives. It is the analytic workhorse behind FDA-mandated safety assessments (including REMS effectiveness assessments), EMA imposed and voluntary PASS, and sponsor-initiated safety monitoring. The design is concrete and quantitative: it builds a cohort, defines exposure and an outcome algorithm, accrues person-time, and produces an incidence rate (or a comparative rate/hazard) with a confidence interval — not a vague "evaluation."

Core conceptual distinction

(the core estimand distinction). Three things are routinely conflated and must be kept separate. (1) Active surveillance (this design) vs spontaneous reporting / disproportionality. A risk evaluation study has a defined denominator (person-time at risk) and can estimate an incidence rate; FAERS/EudraVigilance disproportionality (PRR, ROR, EBGM) has no denominator and can only flag a signal relative to other drugs — it is hypothesis-generating, not risk-quantifying. (2) Absolute risk characterization vs comparative effect estimation. Some PASS deliverables are descriptive (the absolute rate of the AE among initiators, for labeling and benefit-risk); others are causal contrasts (rate in users of drug A vs an active comparator), which require the full confounding-control machinery of comparative-effectiveness designs. The estimand must be pre-specified: a single-arm incidence rate, a rate ratio vs an active comparator, or a within-person relative incidence (self-controlled). (3) Drug-risk evaluation vs risk-minimization-measure (RMM) evaluation. The former measures the AE rate; the latter measures whether an intervention (a Medication Guide, prescriber certification, pregnancy-prevention program) changed knowledge, behavior, drug utilization, or the AE rate — process and outcome indicators, often survey-based for process metrics and claims-based for utilization/outcome metrics. A protocol should state which it is doing and not blur the two.

Pros, cons, and trade-offs

(specific and comparative, naming the alternatives). - vs spontaneous-report disproportionality (FAERS/EudraVigilance signal detection): The cohort risk evaluation study has a real denominator, so it estimates rates and supports labeling and benefit-risk; it controls (with design + adjustment) for confounding. Cost: it needs a longitudinal data source large enough to accrue exposed person-time and outcome events, and it is slower and more expensive than mining a spontaneous-report database. Prefer the cohort study once a signal is credible and a quantified risk is needed; prefer disproportionality only for hypothesis generation across the whole product space. - vs a full comparative-effectiveness / active-comparator new-user (ACNU) study: A single-arm incidence-rate risk evaluation is simpler and answers "how often does this AE occur in users?" Cost: with no comparator, it cannot separate drug effect from background rate, and an external/historical rate is vulnerable to differences in case ascertainment and population mix. Prefer single-arm for absolute risk characterization and rare, drug-specific events with no plausible background; prefer an active comparator the moment the question becomes "does this drug raise risk relative to an alternative," where confounding by indication would otherwise dominate (see `active-comparator-new-user`). - vs self-controlled designs (SCCS, case-crossover): Self-controlled analyses cancel all time-invariant confounding by using the person as their own control and are excellent for transient exposures and acute outcomes. Cost: they require a within-person exposure contrast and assume the outcome does not affect future exposure and that the event does not censor observation. Prefer self-controlled when time-fixed confounding is severe and the exposure is intermittent (e.g., short courses); prefer the cohort for sustained exposures, cumulative-dose questions, and when absolute rates are the deliverable (see `self-controlled-case-series`, `case-crossover`).

When to use

(clear decision rules). A credible safety signal needs to be quantified in routine care; an FDA REMS effectiveness assessment or EMA PASS protocol must deliver an incidence rate, a comparative rate, or a utilization/behavior metric; benefit-risk or labeling decisions require an absolute AE rate among real-world initiators; a regulator has imposed post-marketing surveillance with a denominator-based endpoint. Use the cohort form when exposure is sustained and the AE accrues over follow-up; use a self-controlled or case-only form when time-fixed confounding is the dominant threat and exposure is transient.

When NOT to use — and when it is actively misleading or dangerous

(clear decision rules). - Spontaneous-report-only "evaluation." Running disproportionality on FAERS/EudraVigilance and calling it a risk evaluation produces a signal with no denominator; reporting a PRR/ROR as if it were a risk is wrong and can drive over-reaction or false reassurance. That is signal detection (see `signal-detection`), not risk quantification. - No usable denominator / no observable washout. If a large share of person-time is Medicare Advantage-only (capitated encounters not adjudicated as fee-for-service claims), the exposed denominator and the "no prior event/exposure" washout are partly missing, not zero — incidence rates are then biased by undercount of both numerator and denominator. Restrict to fully observable enrollment. - Immortal time in enrollment/procedure-anchored cohorts. If follow-up (or "exposure") starts at REMS enrollment, certification, or a procedure but the AE clock is allowed to run before the patient could possibly have the event, the immortal interval deflates the rate. Align time zero to first exposure and start follow-up there (see `immortal-time-bias-handling`, `time-zero-index-date-alignment-rwe`). - Differential competing risks by exposure. In elderly or seriously ill populations, death and treatment cessation compete with the AE; if these differ by exposure, naive incidence rates and crude comparisons mislead. Use cause-specific rates or subdistribution methods and report the competing event (see `competing-risks-cause-specific-fine-gray-rwe`). - Single-arm comparison to an unmatched external rate when ascertainment differs — a registry that adjudicates events vs claims that infer them will not produce comparable rates; the contrast is an artifact of case-finding, not biology.

Data-source operational depth

(claims vs EHR vs registry vs linked). - Administrative claims (FFS / commercial): Exposure is the pharmacy claim (NDC + `fill_date` + `days_supply`) or a medical-benefit administration (J-code) for infused products; outcomes are dx/procedure codes with a validated algorithm (a claims phenotype with known PPV). Require continuous medical + pharmacy enrollment across the washout and follow-up so "no prior event" and the denominator are real. Failure modes: MA-only person-time lacks adjudicated FFS claims, so both numerator and denominator are undercounted — exclude MA-only spans or restrict to A/B/D enrollees; sample fills, 90-day mail-order, and stockpiling distort `days_supply` and on-treatment risk windows; differential competing risks by exposure in elderly claims bias crude rates; an AE algorithm with modest PPV inflates the numerator unless calibrated (see `claims-outcome-algorithm-ppv-sensitivity-rwe`). - EHR: Initiation is the order or administration, not the dispensing; linkage to fills confirms the patient actually started. Labs, vitals, and notes sharpen AE ascertainment and severity (an advantage for outcomes like hepatotoxicity or QT), but visit-driven capture means patients who leave the system are differentially lost — define observation windows and treat loss to follow-up as potentially informative; out-of-network events are simply unseen. - Registry: Strongest for adjudicated outcomes, indication, and severity (the numerator is high-quality); typically weak for complete drug exposure and full person-time denominator. Link to claims for fills and to a death index to firm up censoring and capture fatal AEs. - Linked claims–EHR–vital records: The ideal substrate — EHR/registry-grade AE ascertainment + claims-grade exposure and denominator + reliable mortality — but linkage introduces selection (only the linkable subset) and order/fill/service date discrepancies that must be reconciled before time-zero and risk-window assignment.

Worked claims example (absolute incidence of an AE among new users)

Question: incidence of acute severe hepatic injury among adult new users of a newly marketed oral drug under a REMS, in a commercial + Medicare FFS database. (1) Eligibility: age ≥18 and 365 days of continuous A/B/D (or commercial medical+pharmacy) enrollment before the first study fill — this is what makes the denominator observable and the washout real. (2) Washout / incident-user restriction: no fill of the study drug in the 365-day lookback (incident use) and no diagnosis of the AE in the lookback (so the event is incident, not prevalent). (3) Time zero = the first qualifying `fill_date`; do not start the AE clock at REMS enrollment or certification, which would create immortal time. (4) Outcome: first qualifying AE using a validated algorithm (e.g., ≥1 inpatient dx in the primary position, or 1 inpatient + 1 outpatient `dx_code` within a 30-day window) with a known PPV; if PPV is modest, plan a chart-validation substudy or quantitative-bias correction. (5) Follow-up: from time zero to the first AE, censoring at disenrollment, death, end of data, and — for an on-treatment risk window — the last `days_supply` end plus a pre-specified grace period; treat death as a competing risk and report cause-specific person-time. (6) Estimate: events ÷ person-years gives the incidence rate; an exact Poisson 95% CI is appropriate when events are few. (7) Sensitivity: vary the washout and grace period, swap the AE algorithm (high-PPV vs high-sensitivity), and add a negative-control outcome to detect residual surveillance/ascertainment bias. The same scaffold extends to a comparative risk evaluation by adding an active comparator and propensity-score balancing, or to an RMM-effectiveness evaluation by replacing the AE numerator with a utilization/behavior metric (e.g., proportion of dispensings preceded by the required monitoring lab).

Worked example

Scenario

Isotretinoin (a severe-acne drug) carries a known risk of birth defects. The FDA requires a REMS called iPLEDGE: female patients of childbearing potential must use two forms of contraception, pass a monthly pregnancy test, and their pharmacist must confirm a negative result before dispensing each prescription. A safety team wants to know whether pharmacies are actually checking the pregnancy test before each fill — a process metric — and whether pregnancies among drug users are declining — an outcome metric. They use one year of commercial insurance claims for 500 female patients aged 15-45 who filled isotretinoin for the first time.

Dataset

Pharmacy claims table showing isotretinoin fills and required pregnancy-test lab claims for five example patients.

Steps

• Identify the risk: isotretinoin causes serious birth defects, so the REMS requires a negative pregnancy test within 30 days before each monthly fill.

• Identify the mitigation: restricted distribution through iPLEDGE — pharmacies must confirm a negative pregnancy-test lab result in the patient's claims before dispensing.

• Measure process compliance: for each of the 500 fills, look back 30 days in the lab claims table for a pregnancy-test result; a fill with a test present is compliant, a fill without is not.

• From the five example rows: 3 of 5 fills have a documented pregnancy test in the prior 30 days; 2 do not (PT-002 and PT-005).

• Scale to the full cohort: if 430 of 500 fills have a prior pregnancy test, the monitoring-adherence rate is 430 divided by 500 equals 0.86, or 86 percent.

• Measure the outcome: search each patient's claims for a pregnancy diagnosis code (ICD-10 Z34.xx) in the 9 months following any fill; count pregnancies per 100 patient-years on drug.

• In the full cohort, 6 pregnancies are identified across 420 total patient-years of follow-up; the pregnancy incidence rate is 6 divided by 420 equals 0.014 pregnancies per patient-year, or 1.4 per 100 patient-years.

• Compare to the pre-REMS historical rate of roughly 3.0 per 100 patient-years to assess whether the program is reducing pregnancies.

Result

Process metric: 86% of fills had a documented pregnancy test in the prior 30 days (430 of 500), meaning 14% of dispensings did not meet the REMS monitoring requirement. Outcome metric: pregnancy incidence rate of 1.4 per 100 patient-years, compared to a historical pre-REMS rate of approximately 3.0 per 100 patient-years — a roughly 50% reduction — suggesting the mitigation program is working, though incomplete monitoring compliance leaves room for improvement.

Runnable example

import pandas as pd
import numpy as np
from scipy.stats import chi2

WASHOUT_DAYS = 365   # observable, drug-free lookback that defines an incident user
GRACE_DAYS   = 30    # on-treatment risk window = last days_supply end + grace

def build_risk_eval(rx, enroll, ae, death, end_of_data):
    rx = rx.sort_values(["person_id", "fill_date"])

    # Time zero = first fill of the study drug.
    first = (rx[rx["is_study_drug"]]
             .groupby("person_id")["fill_date"].min()
             .reset_index(name="index_date"))

    # Incident-user washout: no study-drug fill in the WASHOUT_DAYS before index.
    prior = rx[rx["is_study_drug"]].merge(first, on="person_id")
    had_prior = prior[(prior["fill_date"] < prior["index_date"]) &
                      (prior["fill_date"] >= prior["index_date"] - pd.Timedelta(days=WASHOUT_DAYS))]
    first = first[~first["person_id"].isin(had_prior["person_id"])].copy()

    # Continuous, FFS-observable enrollment across the full washout through index (no MA-only spans).
    e = enroll.merge(first, on="person_id")
    covers = e[(e["enroll_start"] <= e["index_date"] - pd.Timedelta(days=WASHOUT_DAYS)) &
               (e["enroll_end"]   >= e["index_date"]) & (~e["ma_only"])]
    first = first[first["person_id"].isin(covers["person_id"])].copy()

    # On-treatment exposure end = last study fill's days_supply end + grace.
    rx_idx = rx[rx["is_study_drug"]].merge(first[["person_id"]], on="person_id")
    rx_idx["supply_end"] = rx_idx["fill_date"] + pd.to_timedelta(rx_idx["days_supply"], unit="D")
    tx_end = (rx_idx.groupby("person_id")["supply_end"].max()
              .add(pd.Timedelta(days=GRACE_DAYS)).reset_index(name="tx_end"))
    coh = first.merge(tx_end, on="person_id")

    # Administrative censoring: enrollment end, death, end of data, treatment end.
    enroll_end = enroll.groupby("person_id")["enroll_end"].max().reset_index(name="enr_end")
    coh = coh.merge(enroll_end, on="person_id").merge(death, on="person_id", how="left")
    coh["death_date"] = coh["death_date"].fillna(pd.Timestamp.max)
    coh["censor_date"] = coh[["tx_end", "enr_end", "death_date"]].min(axis=1).clip(upper=end_of_data)

    # First INCIDENT AE strictly after time zero and on/before censoring.
    a = ae.merge(coh[["person_id", "index_date", "censor_date"]], on="person_id")
    a = a[(a["event_date"] > a["index_date"]) & (a["event_date"] <= a["censor_date"])]
    first_ae = a.groupby("person_id")["event_date"].min().reset_index(name="ae_date")
    coh = coh.merge(first_ae, on="person_id", how="left")

    coh["had_event"] = coh["ae_date"].notna().astype(int)
    # Person-time stops at the AE if it occurred, else at administrative censoring.
    stop = coh["ae_date"].fillna(coh["censor_date"])
    coh["person_days"] = (stop - coh["index_date"]).dt.days.clip(lower=0)

    events = int(coh["had_event"].sum())
    py = coh["person_days"].sum() / 365.25
    rate = events / py * 1000.0
    # Exact (Garwood) Poisson 95% CI on the count, scaled to the same denominator.
    lo = 0.0 if events == 0 else chi2.ppf(0.025, 2 * events) / 2
    hi = chi2.ppf(0.975, 2 * events + 2) / 2
    return {"events": events, "person_years": py,
            "rate_per_1000py": rate,
            "ci95": (lo / py * 1000.0, hi / py * 1000.0)}

library(data.table)
WASHOUT_DAYS <- 365L
GRACE_DAYS   <- 30L

build_risk_eval <- function(rx, enroll, ae, death, end_of_data) {
  setDT(rx); setDT(enroll); setDT(ae); setDT(death)
  setorder(rx, person_id, fill_date)

  # Time zero = first study-drug fill.
  first <- rx[is_study_drug == TRUE, .(index_date = fill_date[1L]), by = person_id]

  # Incident-user washout: no study-drug fill in the washout window before index.
  pr <- merge(rx[is_study_drug == TRUE], first, by = "person_id")
  prior_ids <- unique(pr[fill_date < index_date &
                         fill_date >= index_date - WASHOUT_DAYS, person_id])
  first <- first[!person_id %chin% prior_ids]

  # Continuous FFS-observable enrollment across the full washout through index.
  e <- merge(enroll, first, by = "person_id")
  ok <- e[enroll_start <= index_date - WASHOUT_DAYS &
          enroll_end   >= index_date & ma_only == FALSE, unique(person_id)]
  first <- first[person_id %chin% ok]

  # On-treatment end = last days_supply end + grace.
  ri <- merge(rx[is_study_drug == TRUE], first[, .(person_id)], by = "person_id")
  ri[, supply_end := fill_date + days_supply]
  txe <- ri[, .(tx_end = max(supply_end) + GRACE_DAYS), by = person_id]
  coh <- merge(first, txe, by = "person_id")

  # Administrative censoring.
  enr <- enroll[, .(enr_end = max(enroll_end)), by = person_id]
  coh <- merge(coh, enr, by = "person_id")
  coh <- merge(coh, death, by = "person_id", all.x = TRUE)
  coh[is.na(death_date), death_date := as.Date("9999-12-31")]
  coh[, censor_date := pmin(tx_end, enr_end, death_date, end_of_data)]

  # First incident AE after time zero, on/before censoring.
  a <- merge(ae, coh[, .(person_id, index_date, censor_date)], by = "person_id")
  a <- a[event_date > index_date & event_date <= censor_date]
  fae <- a[, .(ae_date = min(event_date)), by = person_id]
  coh <- merge(coh, fae, by = "person_id", all.x = TRUE)

  coh[, had_event := as.integer(!is.na(ae_date))]
  coh[, stop_date := fifelse(is.na(ae_date), censor_date, ae_date)]
  coh[, person_days := pmax(as.integer(stop_date - index_date), 0L)]

  events <- sum(coh$had_event)
  py <- sum(coh$person_days) / 365.25
  ci <- poisson.test(events, py)$conf.int
  list(events = events, person_years = py,
       rate_per_1000py = events / py * 1000,
       ci95 = c(ci[1] * 1000, ci[2] * 1000))
}