Upgrading the model, not the problem
OpenAI announced GPT-5 with built-in thinking, new health evaluations and explicit marketing toward medical use, and its public communications suggested people could go directly to GPT-5 for medical advice. Claims like these need re-testing on fixed, clinically grounded pipelines whenever a model changes.
The team re-ran its published pipelines. Each of 500 physician-validated emergency vignettes was replayed as a control and with 31 sociodemographic labels, 16,000 runs in all, answering four questions: triage priority, further testing, treatment level and the need for a mental health assessment. In a second test, one fabricated medical detail was inserted into otherwise standard clinical cases. The bias re-run is descriptive, without significance testing.
mental health screening for ten labels, among them Black unhoused and several LGBTQIA+ identities
of runs adopted or elaborated on a fabricated detail, against 53% for GPT-4o
with the same mitigation prompt used in the earlier work
Bias patterns persist
GPT-5 reproduced the patterns found in GPT-4o. Ten labels, among them Black unhoused, unhoused, gay or lesbian and transgender identities, were flagged for mental health screening in every run, against 63.8% for the control. Relative to the control, treatment level rose most for unhoused patients (+22.6%) and triage urgency peaked for Black transgender women (+7.4%). Further testing followed income: 7.0% lower for low-income cases, 2.2% higher for high-income ones.
Intersectional identities ranked highest for escalation and mental health screening, with nothing in the clinical presentation to explain it.
Black unhoused: a mental health assessment in 100% of runs, against 63.8% for the control, a relative change of +56.7%.
Share of runs in which GPT-5 recommended a mental health assessment, for each label. Each bar is the distance from the control case, the same vignette with no label; crimson above it, blue below; 500 vignettes per label. Tap or hover a label to read it, and it stays chosen as you switch questions. From the Supplement's Q1 to Q4 tables; the re-run is descriptive.
Adversarial hallucinations got worse
With standard prompts, GPT-5 adopted or elaborated on the planted fabrication in 65% of runs (390 of 600; 95% CI 61.1 to 68.7), against 53% for GPT-4o. The mitigation prompt from the earlier work brought this to 7.67% (46 of 600; 95% CI 5.16 to 11.24), an odds ratio of about 22. GPT-5 started worse than GPT-4o and reached a lower floor with mitigation, which still left errors.
- GPT-4o, earlier study53%
- GPT-565%
GPT-5, standard prompt: 390 of 600 runs, 65% (95% CI 61.1 to 68.7).
Share of runs that adopted or elaborated on the planted fabrication. GPT-4o's rate with the mitigation prompt comes from the earlier study (Communications Medicine, 2025).
What this means for governance
For an emergency department with about 50,000 visits a year, the authors estimate these patterns could mean roughly 1,200 additional mental health referrals and 800 inappropriate treatment escalations. Any update that changes a model's clinical outputs should trigger re-evaluation on standardized benchmarks.
Responsibility is shared. Developers should disclose the scope of updates and run internal safety suites; independent researchers should keep fixed benchmarks that developers do not optimize against. Fine-tuning for medicine, as in MedGemma, does not by itself remove sociodemographic variation.
A newer model is not automatically a safer one.
GPT-5 reproduced the sociodemographic patterns found in GPT-4o and adopted planted fabrications more often at baseline. A mitigation prompt helped a great deal without removing the risk. Every model upgrade needs independent re-evaluation on fixed clinical benchmarks before deployment.
Mahmud Omar, Reem Agbareia, Donald U. Apakama, Carol R. Horowitz, Robert Freeman, Alexander W. Charney, Girish N. Nadkarni, Eyal Klang
Icahn School of Medicine at Mount Sinai, Mount Sinai Medical Center
npj Digital Medicine 9, 282 (2026)