Models that assess the "seriousness" of adverse events (AEs), trained on 242,312 FAERS reports for GLP-1 receptor agonists, have published ROC-AUCs between 0.877 and 0.930. Post hoc calibration made their probabilities even more reliable [1]. Strong numbers, but I don't think they measure what the safety team will be tempted to use them for.

Those AUCs mostly show that a model can learn how reporters classify AE seriousness. They don't show which GLP-1 risks are clinically serious.

"Cases were labeled serious if any outcome code (DE, LT, HO, DS, CA, RI, OT) was present." [2] That's a box ticked at reporting time. Nobody adjudicated it. AE seriousness is a regulatory construct with seven categories, the last being the catch-all "important medical event", yet models in this literature learn from labels set by pharmacovigilance experts, not from adjudicated clinical outcomes [3].

Now look at what the model leaned on. SHAP (SHapley Additive exPlanations) put reporter type among the most influential predictors, next to dosing/administration issues, injection-site reactions, pancreatic effects, and temporal variables [2]. Molecule identity, the one feature carrying drug-specific risk, contributed only modestly [2]. The authors themselves flag a disconnect between reporting frequency and predictive importance, with common gastrointestinal events adding little [1]. I suspect that says more about who files and codes a case than about which events actually hurt patients. (Reporter type outranking the molecule should make any safety physician wince.)

The fair objection

AE seriousness IS the regulatory target. Expedited reporting depends on that determination, so a model that reproduces expert coding correctly is doing its job. For case processing, this makes sense. Faster routing of incoming reports is a legitimate use, and demands clinical truth. Even the leading applied study says AE seriousness classification "will always require human confirmation" [3]. That's the right posture for a triage tool.

The trouble starts when the same AUC gets read as evidence for ranking which GLP-1 signals deserve investigation. Ranking risk is a different job, and the robustness data are ugly. In a GLP-1 pancreatitis prioritization study, gradient boosting hit an AUROC of 0.884 on a stratified random split, then dropped to 0.613 under temporal validation [4]. The same study found disproportionality analysis the most stable method for signal detection [4].

Leakage, when the training data contains the answer the model is supposed to predict, artificially boosts model performance. A leakage-controlled triage study reports that published FAERS seriousness performance is "frequently inflated by data label leakage and by evaluation designs that do not reflect prospective use" [5]. Its clinical-only classifier slid from a temporal AUROC of 0.896 to 0.740 on novel drugs held out of training [5]. Novel drugs are where a signal program needs help most, and where these models are weakest. Though these temporal AUROC with novel drugs are probably a more realistic estimate of model performance.

So a 0.930 tells me the algorithm copied the coders well. Whether it found the dangerous drug-event pairs is a question that remains unanswered.

If you run or review pharmacovigilance and RWE work, insist on temporal validation with new drugs held out before any AE seriousness AUC goes near signal prioritization. Then test the model prospectively on live case flow, and score it a second time against adjudicated clinical outcomes, not the reporter-assigned code. Leave disproportionality analysis and human reviewers in charge of what gets investigated. For now...