A particular marketing claim has propped up healthcare AI for half a decade: that a tool built specifically for medicine is, by virtue of that specialization, more trustworthy than a general-purpose chatbot. A June 2026 Nature Medicine study just put a large dent in it.[1]

Researchers benchmarked three frontier general-purpose models — OpenAI's GPT-5.2, Google's Gemini 3.1 Pro Preview, and Anthropic's Claude Opus 4.6 — against purpose-built clinical products including OpenEvidence and UpToDate's Expert AI.[1] The evaluation ran in three stages: 500 MedQA questions (medical knowledge), 500 HealthBench items (alignment with clinician judgment), and a real-clinical-query benchmark built from 100 de-identified questions physicians actually asked in a live clinical setting, scored by 12 US clinicians in randomized, blinded review across roughly 1,800 annotations.[1,2]

The result was not close-but-mixed. The general-purpose models won all three stages, and the clinicians preferred their answers.[1,2] The detail that should sting the specialized-tool vendors: on the real-physician-query benchmark, the dedicated clinical tools performed comparably to an auto-enabled Google Search AI Overview.[2] A product marketed as a clinical-grade reasoning engine scored about like the thing that appears above your search results.

The takeaway is not "specialization is dead"

Read carelessly, this is a story about general models eating clinical AI. Read carefully, it is a story about where the defensible advantage actually lives. The frontier models were not fine-tuned on proprietary clinical corpora; they were just very good general reasoners. That tells you the moat was never the narrowness of the training data. The moat — if there is one — is the evidence system wrapped around the model: workflow fit, retrieval with real provenance, validation design, monitoring, licensing, and a defensible answer to "who owns the mistake when this is wrong."

For an HEOR or RWE audience this should feel familiar, because it is the discipline we already apply to evidence. You do not evaluate a data source in the abstract; you evaluate it against a question — population, use case, comparator, endpoints, operating context, subgroup risks, monitoring plan, and the consequence of a wrong decision. A clinical AI tool is an evidence-generating instrument. Evaluate it like one.

Four critiques to hold alongside the headline

Because this is RWEdnesdays and not a press release, the caveats matter as much as the finding.

Benchmarks are not bedside. MedQA and HealthBench are knowledge and alignment proxies. Even the real-query benchmark is retrospective scoring of answers, not a prospective measure of whether a patient was helped or harmed. Winning these evaluations is necessary; it is nowhere near sufficient for deployment.

The test set may be home turf. The real-query benchmark was built from questions clinicians asked inside a general-purpose-LLM environment. That distribution reflects how clinicians already use general models, which can quietly tilt the test toward them. Call it a construct-validity problem, not a footnote.

"Comparable to Google AI Overview" is double-edged. It is a damning line for vendors. But it also reveals what the benchmark rewards: fluent, well-organized synthesis. The things specialized tools are supposed to optimize for — citation discipline, licensed-source grounding, auditability, regulatory posture, liability — are precisely the things a preference-and-accuracy benchmark measures poorly. The tool may have been scored on the wrong job.

Preference is not correctness — and almost nothing here is regulated. Blinded clinician preference can reward confident, tidy answers over calibrated uncertainty — a known failure mode of fluent models. And the regulatory picture is not what the headlines imply: neither the general-purpose models nor the specialized tools in this study are FDA-cleared medical devices. OpenEvidence and UpToDate Expert AI operate as non-device clinical decision support — they show their sources and rationale so a clinician independently reviews them, which keeps them outside FDA device clearance under the 21st Century Cures Act.[3] So the validation gap is not that the wrong tools got cleared; it is that essentially none of them are evaluated as regulated clinical-performance devices. The market competes on benchmarks and adoption, not on decision-grade evidence.

Where this lands for evidence leaders

The trust claim is shifting under our feet. "Built for medicine" is no longer a credential; it is a marketing adjective. The credential that replaces it is specific and boring: validated for this decision, in this workflow, for this population, against the right comparator, with a monitoring plan attached.

That reframing has teeth in procurement and HTA. If you buy or assess clinical AI, stop accepting benchmark scores and adoption numbers as evidence of fitness. Ask for deployment-context validation: where was this tested, against what reference standard, for which patients, with what error rate, monitored how. The vendors who can answer that will win the next cycle regardless of whether their model is "specialized" — and the ones who cannot will keep scoring like a search box.

Healthcare does not need smaller AI ambition. It needs better evidence discipline around increasingly general AI capability.