There is a number that should be keeping systematic review vendors up at night, and it is not a price. It is 81.7% — the screening sensitivity of traditional dual human review when measured against a rigorous reference standard in the otto-SR evaluation. The LLM-based agentic workflow it was compared against hit 96.7%.[1] Data extraction told the same story: 93.1% accuracy for the machine, 79.7% for the humans.[1]
Sit with the implication. For two decades, "dual independent human review" has been the methodological incantation that made a literature review systematic — the thing PRISMA-trained reviewers and Cochrane handbooks treated as the floor of credibility. It turns out the floor has measurable error rates, and a well-orchestrated set of language-model agents can beat them. The gold standard was always just a comparator nobody had benchmarked properly.
The frontier has since moved past task-level assistance entirely. A February 2026 preprint described a fully automated pipeline — literature search through completed manuscript, no human in the loop — processing hundreds of papers through iterative inclusion, extraction, and synthesis calls, with citation accuracy of 95.87%.[2] You can argue (I would) that a zero-human systematic review is a stunt with an unsettling publication-ethics tail. But as an existence proof it does its job: every individual step of the SR pipeline is now automatable at human-competitive quality.
What the cautious reviews get right — and what they miss
The methodological literature is responding with appropriate caution. A 2026 systematic review of LLM performance across SR tasks finds promising results but calls for careful implementation,[3] and a widely cited scoping review concludes the technology is "on the rise, but not yet ready" for unsupervised use.[4] Both flag real failure modes: inconsistent performance across review types, sensitivity to prompting, poorly understood behavior when synthesizing across heterogeneous study designs and risk-of-bias profiles. All true. All important.
But the cautious framing smuggles in a false baseline — it benchmarks the machines against an idealized human process rather than the actual one, with its 81.7% sensitivity, its fatigued second reviewers, its three-month screening backlogs, and its five- to six-figure invoices. "Not yet ready to replace humans" is the wrong question when the humans were never as ready as we pretended.
Where this actually lands
The stable equilibrium is not zero-human reviews, and it is not artisanal hand-screening either. It is verified hybrid: machines do first-pass screening and extraction with full audit logs; humans design the protocol, adjudicate the genuinely hard calls, and — critically — verify against sampled reference standards with pre-specified error tolerances. The deliverable changes from "we followed the ritual" to "here is our measured error rate, and here is why it is acceptable for this decision."
That last clause is the HEOR-shaped insight. Evidence synthesis serves decisions — HTA submissions, guideline panels, payer dossiers — and decisions have error tolerances. A scoping review feeding a research agenda can tolerate more screening error than a clinical effectiveness review feeding a JCA dossier. Pretending one workflow fits all was always lazy; now it is also expensive.
Three predictions, time-stamped June 2026. One: within two years, major HTA bodies will accept (and then expect) machine-assisted screening with reported validation metrics — the reporting-framework groundwork via CONSORT-AI-style extensions is already being laid.[3] Two: the SR vendor market bifurcates into validation-and-protocol shops and a race-to-the-bottom commodity tier, and the middle gets gutted. Three: the first major retraction of an AI-conducted review for a subtle, systematic extraction error happens within 18 months — and it will be caught by another automated pipeline, which is precisely the point.
The review is dead. Long live the protocol.