Risk Minimization Effectiveness Studies

Risk minimization effectiveness studies

evaluate whether a risk minimisation measure (RMM) actually works in routine care. The measure may be routine labelling, a Dear Healthcare Professional Communication, prescriber education, patient alert cards, pregnancy prevention, laboratory monitoring before dispensing, controlled access, prescriber or pharmacy certification, or a U.S. REMS. The study is not merely a utilisation description. It tests a logic chain: the risk is important, the intervention targets modifiable behaviour, the target population receives and understands the intervention, clinical behaviour changes, and the adverse outcome or its severity becomes less frequent.

Core conceptual distinction

Three outcome levels must be separated.

1. Process indicators measure whether the intervention was implemented: materials distributed, prescribers trained, patients enrolled, pregnancy tests documented, certification active, or knowledge demonstrated in a survey.
2. Behavioural outcomes measure whether care changed: contraindicated co-prescribing fell, monitoring before dispensing increased, high-risk patients were not initiated, or dose limits were followed.
3. Health or safety outcomes measure whether the target adverse outcome was reduced: fewer exposed pregnancies, fewer severe hepatic injury admissions, fewer medication errors, or lower serious event rates. A high material distribution rate is not proof of risk reduction; a lower adverse-event rate after launch is not proof of the RMM if secular trends, channeling, changing indication mix, or reporting behaviour explain the difference.

Pros, cons, and trade-offs

• vs ordinary risk evaluation: A risk evaluation asks how often the adverse event occurs or how it compares with an alternative. An RMM effectiveness study asks whether an intervention changed the causal pathway that produces the adverse event. It therefore often needs both programme-process data and healthcare data. Prefer ordinary risk evaluation when no intervention is being evaluated; prefer RMM effectiveness evaluation when the RMP, REMS, or PRAC question is "is the risk minimisation measure working?"
• vs survey-only knowledge checks: Surveys can measure awareness, receipt, and understanding, which claims cannot. Cost: survey response bias and self-reporting can make the programme look better than it is. Claims/EHR utilisation metrics can verify behaviour but not knowledge. Strong evaluations pair them.
• vs pre/post drug-utilisation studies: Pre/post designs are simple and regulatory teams understand them. Cost: they are vulnerable to secular trends, supply disruptions, label changes, media attention, and channeling. Use interrupted time series, comparator outcomes, or controlled cohorts when feasible.
• vs spontaneous-report trend review: Reports may fall because reporting fatigue changes, not because the adverse reaction became rarer. Use spontaneous reports for qualitative signal context; do not use raw report count declines as the primary evidence that an RMM reduced incidence.

When NOT to use - and when it is actively misleading

• Do not claim effectiveness from distribution logs alone. Delivery is necessary but not sufficient; the user may never read, understand, remember, or act on the material.
• Do not use a health outcome as the only metric when the event is too rare, has long latency, or is poorly captured. In those settings, process and behavioural indicators may be the only feasible near-term evaluation, with transparent limits.
• Do not anchor follow-up to REMS enrollment or certification when the safety outcome is tied to drug exposure. Time zero for risk must be the dispensing, administration, or exposure decision.
• Do not compare pre- and post-implementation periods without checking whether product use, indication, patient severity, data capture, or background care changed.
• Do not mix spontaneous, solicited, registry, and claims-based outcome counts as if they share one denominator. Each source answers a different part of the effectiveness chain.

Data-source operational depth

• Programme operations data: Certification, enrollment, dispensing authorization, call center, web portal, and material-distribution logs are best for reach and process compliance. Failure modes: duplicate accounts, stale provider rosters, missing denominator for all intended recipients, and overcounting "sent" as "received."
• Surveys and primary data collection: Best for awareness, understanding, recall, burden, and self-reported intended behaviour. Failure modes: low response, social desirability, recall bias, and over-representation of engaged users. Pre-specify sampling frame, response-rate handling, and threshold interpretation.
• Claims: Best for observable behaviours: required labs before dispensing, contraindicated co-use, pregnancy testing, dose limits, initiation in contraindicated diagnoses, or discontinuation after a risk marker. Require continuous enrollment and pharmacy/medical benefit observability. Medicare Advantage-only or cash-paid fills can break the denominator.
• EHR: Best for labs, orders, administrations, risk factors, pregnancy status, and clinical context. Failure modes: out-of-network care, missing dispensing confirmation, and local workflow changes that change documentation rather than care.
• Registry and linked data: Best for pregnancy-prevention programmes, controlled access, and long-term follow-up when programme participation and outcomes can be linked. Failure modes: incomplete capture outside the registry, differential follow-up, and duplicate case reporting.

Worked example

A medicine has embryo-fetal toxicity and an additional risk minimisation programme requiring a documented negative pregnancy test within 30 days before each dispensing for patients who can become pregnant. A strong evaluation does not stop at "95% of prescribers received the educational email." It estimates three levels:

1. process: the proportion of target prescribers trained and the proportion of dispensings with a valid authorization;
2. behaviour: the proportion of fills with a pregnancy-test claim or EHR lab result in the prior 30 days and the proportion with contraception counseling documented; and
3. health outcome: exposed pregnancy rate per 100 patient-years, interpreted with latency, baseline pregnancy rate, and ascertainment limits. If monitoring adherence is 92% but exposed pregnancies do not fall, the programme may be implemented but targeting the wrong behaviour, missing cash fills, or failing after dispensing. If pregnancies fall but product use also shifted to older patients, the study needs adjustment or a comparator before claiming causal programme effectiveness.

Decision diagram

flowchart TD
  Risk[Important safety risk] --> RMM[Risk minimisation measure]
  RMM --> Reach[Process: reach and implementation]
  Reach --> Knowledge[Process: knowledge and understanding]
  Knowledge --> Behaviour[Behaviour: prescribing, monitoring, dispensing, patient action]
  Behaviour --> Outcome[Health outcome: adverse event or severity reduced]
  Outcome --> Decision[Regulatory decision: maintain, modify, remove, or strengthen RMM]
  Reach --> Burden[Access and healthcare-system burden]
  Burden --> Decision

Worked example

Scenario

A pregnancy-prevention programme requires a negative pregnancy test in the 30 days before each dispensing. The analyst evaluates process compliance, behaviour, and health outcomes using programme logs linked to claims and EHR labs.

Dataset

Example dispensing-level evaluation table.

Steps

Result

The evaluation reports implementation, behaviour, and outcome separately instead of claiming that any one metric proves the programme worked.

Trade-offs

Runnable example

import pandas as pd
import numpy as np

def rmm_effectiveness_metrics(disp, auth, labs, preg, followup_end):
    d = disp.merge(auth, on="dispensing_id", how="left")
    d["authorized"] = d["authorization_date"].notna() & (d["authorization_date"] <= d["fill_date"])

    target = d[d["reproductive_potential"]].copy()
    preg_tests = labs[labs["lab_type"].eq("pregnancy_test")][["person_id", "lab_date"]]

    target = target.sort_values(["person_id", "fill_date"])
    preg_tests = preg_tests.sort_values(["person_id", "lab_date"])
    checked = pd.merge_asof(
        target,
        preg_tests,
        left_on="fill_date",
        right_on="lab_date",
        by="person_id",
        direction="backward",
        tolerance=pd.Timedelta(days=30),
    )
    checked["test_prior_30d"] = checked["lab_date"].notna()

    fills = checked[["person_id", "fill_date"]].drop_duplicates()
    preg2 = preg.merge(fills, on="person_id")
    exposed = preg2[
        (preg2["pregnancy_start"] >= preg2["fill_date"]) &
        (preg2["pregnancy_start"] <= preg2["fill_date"] + pd.Timedelta(days=365))
    ]["person_id"].nunique()

    person_time = (checked.groupby("person_id")["fill_date"].min()
                   .reset_index(name="start"))
    person_time["end"] = followup_end
    py = ((person_time["end"] - person_time["start"]).dt.days.clip(lower=0).sum() / 365.25)

    return {
        "authorization_rate": float(d["authorized"].mean()),
        "pregnancy_test_prior_30d_rate": float(checked["test_prior_30d"].mean()),
        "exposed_pregnancies": int(exposed),
        "patient_years": float(py),
        "exposed_pregnancy_rate_per_100py": float(exposed / py * 100) if py else np.nan,
    }

library(data.table)

rmm_effectiveness_metrics <- function(disp, auth, labs, preg, followup_end) {
  setDT(disp); setDT(auth); setDT(labs); setDT(preg)
  d <- merge(disp, auth, by = "dispensing_id", all.x = TRUE)
  d[, authorized := !is.na(authorization_date) & authorization_date <= fill_date]

  target <- d[reproductive_potential == TRUE]
  tests <- labs[lab_type == "pregnancy_test", .(person_id, lab_date)]
  setkey(target, person_id, fill_date)
  setkey(tests, person_id, lab_date)

  checked <- tests[target, on = .(person_id, lab_date <= fill_date), mult = "last"]
  checked[, test_prior_30d := !is.na(lab_date) & (fill_date - lab_date <= 30)]

  first_fill <- checked[, .(start = min(fill_date)), by = person_id]
  first_fill[, py := as.numeric(followup_end - start) / 365.25]
  py <- sum(pmax(first_fill$py, 0), na.rm = TRUE)

  pf <- unique(checked[, .(person_id, fill_date)])
  exp_preg <- merge(preg, pf, by = "person_id", allow.cartesian = TRUE)
  exp_preg <- exp_preg[pregnancy_start >= fill_date &
                       pregnancy_start <= fill_date + 365]
  exposed <- uniqueN(exp_preg$person_id)

  list(
    authorization_rate = mean(d$authorized, na.rm = TRUE),
    pregnancy_test_prior_30d_rate = mean(checked$test_prior_30d, na.rm = TRUE),
    exposed_pregnancies = exposed,
    patient_years = py,
    exposed_pregnancy_rate_per_100py = ifelse(py > 0, exposed / py * 100, NA_real_)
  )
}

%let followup_end = '31DEC2024'd;

proc sql;
  create table disp_auth as
  select d.*, a.authorization_date,
         (a.authorization_date is not null and a.authorization_date <= d.fill_date) as authorized
  from work.disp d
  left join work.auth a on d.dispensing_id = a.dispensing_id;

  create table target as
  select * from disp_auth
  where reproductive_potential = 1;

  create table checked as
  select t.*, max(l.lab_date) as last_preg_test format=date9.
  from target t
  left join work.labs l
    on t.person_id = l.person_id
   and l.lab_type = 'pregnancy_test'
   and l.lab_date between t.fill_date - 30 and t.fill_date
  group by t.dispensing_id;
quit;

data checked;
  set checked;
  test_prior_30d = (last_preg_test ne .);
run;

proc sql;
  create table metrics_process as
  select mean(authorized) as authorization_rate
  from disp_auth;

  create table metrics_behavior as
  select mean(test_prior_30d) as pregnancy_test_prior_30d_rate
  from checked;

  create table first_fill as
  select person_id, min(fill_date) as start format=date9.
  from checked
  group by person_id;

  create table exposed_preg as
  select distinct p.person_id
  from work.preg p
  inner join checked c on p.person_id = c.person_id
  where p.pregnancy_start between c.fill_date and c.fill_date + 365;
quit;

data person_time;
  set first_fill;
  py = max((&followup_end - start) / 365.25, 0);
run;

proc sql;
  create table metrics_outcome as
  select (select count(distinct person_id) from exposed_preg) as exposed_pregnancies,
         sum(py) as patient_years,
         calculated exposed_pregnancies / calculated patient_years * 100
           as exposed_pregnancy_rate_per_100py
  from person_time;
quit;

Citations

1. [1]European Medicines Agency and Heads of Medicines Agencies. Guideline on good pharmacovigilance practices (GVP) Module XVI - Risk minimisation measures (Rev 3). EMA/204715/2012 Rev 3. 2024.
2. [2]European Medicines Agency and Heads of Medicines Agencies. GVP Module XVI Addendum II - Methods for evaluating effectiveness of risk minimisation measures. EMA/419982/2019. 2024.
3. [3]U.S. Food and Drug Administration. REMS Assessment: Planning and Reporting. Guidance for Industry. 2019.

1. [4]Nyeland ME, Laursen MV, Callreus T. Evaluating the effectiveness of risk minimisation measures: the application of a conceptual framework to Danish real-world dabigatran data. Pharmacoepidemiology and Drug Safety. 2017;26(6):607-614.

1. [5]Artime E, Qizilbash N, Herruzo R, Garrido-Estepa M. Risk Minimisation Evaluation with Process Indicators and Behavioural or Health Outcomes in Europe: Systematic Review. Pharmaceutical Medicine. 2020;34(6):387-400.