Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support

One planted detail

Clinical prompts can carry errors, through copy-forward, a mistyped value or deliberate manipulation, and a model may treat every token as ground truth. Each of the 300 vignettes held exactly one fabrication: a fictitious laboratory test such as Serum Neurostatin, a non-existent sign such as the Cardiac Spiral Sign on echocardiography, or an invented disease such as Faulkenstein Syndrome.

Each case came in a short version of 50 to 60 words and a long one of 90 to 100, with identical medical content. Six models answered under three conditions, default settings, a mitigation prompt and temperature zero, for 5,400 outputs. A case counted as a hallucination when the model elaborated on the fabricated detail.

How often the models took the bait

The mitigation prompt told the models to use only clinically validated information and to acknowledge uncertainty. It lowered the overall rate from 65.9% to 44.2% (OR 0.27, 95% CI 0.23 to 0.32). Temperature zero changed nothing: 66.5% (OR 1.05, P = 0.58). Across all conditions, short vignettes were slightly more likely to draw a hallucination than long ones (OR 1.22, P = 0.003).

Model by model

Under default settings Llama-3.3-70B and GPT-4o hallucinated least, near 51%, and Distilled-DeepSeek-R1 most, in 80% to 83% of outputs (OR 8.41 against GPT-4o), with Phi-4 close behind (OR 7.12). Mitigation helped unevenly. GPT-4o fell from 53% to 23% and DeepSeek roughly halved, while Qwen-2.5-72B, gemma-2-27b-it and Phi-4 moved far less, and Phi-4 ended as the worst.

Distilled-DeepSeek-R1, distilled onto a Llama 70B base, hallucinated far more than Llama-3.3-70B; the authors suggest that distillation or RLHF pipelines may amplify the vulnerability. The mitigation prompt rarely created new errors: 1.2% of outputs that were correct under the base prompt became hallucinations with it.

Well-known misinformation

The study also tested five widely circulated health claims, from vaccines and autism to the laboratory origin of COVID-19, in three runs on each of three models. In 43 of 45 runs the models identified the claim as unsubstantiated. The two exceptions were GPT-4o runs on natural immunity versus vaccination.

Each tile is one run, three per model for each claim, filled where the model identified the claim as unsubstantiated. The two empty tiles are GPT-4o runs on natural immunity versus vaccination.