Frequently wrong, never in doubt

Among 60 clinician-authored vignettes spanning 21 clinical domains, evaluated under 16 factorial conditions for a total of 960 responses, ChatGPT Health undertriaged 52% of gold-standard emergencies — directing patients with diabetic ketoacidosis or impending respiratory failure to outpatient evaluation within 24 to 48 hours rather than the emergency department. The finding, published in Nature Medicine on February 23, 2026, is the number that gives Gary Marcus’s most repeated slogan — “frequently wrong, never in doubt” — its weight.

The number is one of four such findings to land between February and April 2026. Three come from models tested alone on written questions or cases, one from lay people using chatbots in a randomized trial, and all point the same way: off-the-shelf models, used without clinician oversight, are unreliable medical assistants. Marcus packaged the four into a slogan on April 21, 2026: “frequently wrong, never in doubt, and that applies in medicine.”

The four studies

The BMJ Open audit, published April 14, 2026, probed five chatbots — Gemini, DeepSeek, Meta AI, ChatGPT, and Grok — across ten questions in five medical domains in February 2025. The headline number is blunt: “Half (50%) the responses were problematic: 30% were somewhat, and 20% were highly problematic.” Grok gave problematic answers 58% of the time (29 of 50), and 30% of its responses (15 of 50) were highly problematic, the highest share among the five. The authors describe the outputs as “authoritative-sounding but potentially flawed” — a phrasing closer to Marcus’s “purveyors of authoritative bullshit,” which he applied to LLMs in medical contexts in the same April 21 post.

The JAMA Network Open paper, published the same month, takes a wider net: 21 frontier models on 29 structured clinical vignettes, 16,254 responses across successive stages of clinical reasoning, with no lay users involved. The authors’ summary is restrained. Models “remain limited in early diagnostic reasoning and cannot yet be relied on for unsupervised patient-facing clinical decision-making.” The qualifier unsupervised is the one the deployment-optimist case rides on.

The triage paper from February 2026 closes the loop on the headline numbers. Sixty clinician-authored vignettes, 21 clinical domains, 16 factorial conditions, 960 responses. The 52% under-triage of gold-standard emergencies is the headline, but the sharper finding is the second one: anchoring statements in the user’s prompt — phrasing that suggests a less-acute condition — raised the rate of triage shifts in edge cases from 3.3% to 13.3%; the paper reports an odds ratio of 11.7 for the anchoring effect. A shift is not necessarily an error: 30 of the 32 anchoring-associated shifts stayed within acceptable clinical bounds. The model is not just wrong on average; it is steerable by how the user frames the case, which in emergency triage is a failure mode that matters.

The asymmetry

The first Nature Medicine study, “Reliability of LLMs as medical assistants for the general public,” is the most methodologically pointed of the four, and its central finding deserves to stand on its own. The authors randomized 1,298 participants between assistance from GPT-4o, Llama 3, Command R+, or a self-chosen source — between August 21 and October 14, 2024 — across ten medical scenarios.

Tested alone, LLMs complete the scenarios accurately, correctly identifying conditions in 94.9% of cases and disposition in 56.3% on average. However, participants using the same LLMs identified relevant conditions in fewer than 34.5% of cases and disposition in fewer than 44.2%, both no better than the control group.

Models alone score 94.9% on conditions. Lay users with the same models score under 34.5%. Most of that 60-point gap sits in the human-LLM interaction loop, though not all of it: the models also misread queries and supplied incorrect information, and on their own they chose the right disposition only 56.3% of the time.

That distinction reframes what “LLMs are unreliable” means in medical settings, and the reframing is more damaging to deployment optimism than a flat accuracy number would have been. Retrieval and post-training may bring down a 50% problematic rate. No amount of training makes lay users prompt better. The models that score 94.9% alone are the same models whose lay users identify the relevant condition in under 34.5% of cases, on the same scenarios, in a randomized trial. The largest failure is in the surface where the weights meet a non-expert under stress.

What the slogan covers, and what it does not

The four studies test deployment scenarios that production systems do not match. Three test off-the-shelf models alone, on generic chatbot Q&A, written clinical vignettes and patient-style triage prompts. The fourth puts lay users at the keyboard, prompting cold, without the structured intake forms or clinician-in-the-loop checkpoints that real clinical decision-support deployments wrap around the model. None of the four gave the models curated clinical knowledge, patient records, or clinician oversight. The strongest counter to Marcus, made implicitly by every health-system partnership announced over the last eighteen months, is that the literature shows LLMs are unreliable when used badly — not that LLMs are unreliable when used as deployed.

Clinical-decision-support vendors have spent years building curated clinical retrieval, structured-data harnesses, and audit trails. The studies supplied none of that. The closest any came was open-web retrieval: the randomized trial included Command R+ for its retrieval-augmented generation, which searches the open internet before answering. The gap between audited consumer chatbots and production clinical systems would be closed by peer-reviewed trials of the production stack: retrieval-augmented models with curated clinical knowledge bases, structured EHR data, clinician oversight loops, and the audit infrastructure that vendors describe in their announcements. None of the four studies on Marcus’s table is such a trial.

Marcus’s slogan does not draw that line cleanly. “Frequently wrong, never in doubt, and that applies in medicine” reads, on the page, as a claim about LLMs in medicine generally. The literature he cites is narrower: it is about off-the-shelf models and consumer chatbots, tested alone or in lay hands, without clinical scaffolding. On a charitable reading — the reading the four studies actually support — Marcus is making a deployment-caution claim about the consumer-product surface, where the slogan is correctly aimed and the numbers carry it. On the uncharitable reading, the slogan does work it has not earned: it is heard as a claim about clinical AI as a category, including the structured deployments the literature does not test.

Whether structured clinical deployments hold up is the next round of papers.

What the record actually shows

Most of Marcus’s contrarian register is rhetorical — “deep learning is hitting a wall,” “the curve has flattened,” “pattern matching, not reasoning.” Whether those phrases are validated depends on which benchmark, which year, which interpretation. The healthcare-deployment claim is different in kind. It rests on four peer-reviewed studies, published in three journals between February and April 2026, with sample designs that can be inspected and effect sizes that can be argued about. Whether the slogan is correctly scoped to consumer-facing deployment, whether it generalizes to clinical decision support, whether the production-stack trials will show the same pattern — these are questions the literature can adjudicate.

Where Marcus’s bet to Amodei imports settlement-ready discipline to a forecast debate, the four studies do the same work on a deployment claim: dated, criteria-bounded and open to inspection. The 52% under-triage figure, the 60-point gap between models-alone and lay-user accuracy, the 11.7 odds ratio for anchoring-induced shift — these are the results a reader can take to a clinical-AI conversation without needing Marcus’s framing.