ChatGPT Health performance in a structured test of triage recommendations

Best in the middle, worst at the edges

ChatGPT Health launched in January 2026 as OpenAI's consumer health tool and has reached millions of users. It gives symptom guidance at first contact, with no clinician in between. To test whether it fails safely, the team submitted 60 clinician-authored vignettes across 21 clinical domains, each under 16 factorial conditions.

Accuracy peaked in the middle of the acuity range: 93.0% for semi-urgent cases and 76.9% for urgent ones. At the extremes, where errors cost the most, it fell. Among true emergencies, 51.6% were under-triaged. Among non-urgent presentations, 64.8% were over-triaged to a scheduled physician visit.

Where it breaks: emergencies that evolve

Four emergency vignettes tested asthma exacerbation and diabetic ketoacidosis (DKA), each with and without objective findings. Asthma exacerbation accounted for 28 of the 33 under-triaged emergency responses (84.8%). The model named the warning sign, an elevated CO2, and then reasoned it away, concluding that the findings did not prove immediate respiratory failure.

In DKA, the model correctly identified early or mild DKA and still recommended outpatient care, as if a condition that is an emergency by definition were simple hyperglycemia. A supplementary set of four textbook emergencies (stroke, anaphylaxis, meningitis, aortic dissection) showed no under-triage. This suggests the model identifies classic presentations but fails when emergency status depends on clinical progression.

The model recognized the warning signs in evolving emergencies, then talked itself out of escalating care.

Anchoring and the power of framing

Of eight pre-specified hypothesis tests, only anchoring changed triage significantly. When family or friends minimized the symptoms, triage shifted in edge cases (OR 11.7, 95% CI 3.7 to 36.6), and 52.5% of the shifts moved toward less urgent care. The effect was confined to ambiguous scenarios; clear-cut cases did not shift.

Patient race, sex and barriers to care had no significant effect on triage, although the confidence intervals were wide. Adding objective findings such as lab values and vital signs raised overall accuracy from 54.6% to 77.9% (OR 9.4, P < 0.001) and cut over-triage of non-urgent cases by 61 percentage points. For emergencies the direction reversed: objective findings increased under-triage by 9.3 percentage points, a difference that was not significant (P = 0.62).

The crisis guardrail problem

The most consequential failure came in the suicidal ideation vignettes. In one, a 27-year-old described thoughts of taking a lot of pills. Without objective findings, the crisis banner linking to the 988 Suicide and Crisis Lifeline appeared in all 16 prompt variants. With normal test results added, at the same clinical severity, it appeared in none.

Across all seven suicidal ideation scenarios (224 responses), the crisis banner fired in only 4 of 14 vignettes, and the other ten produced no safety alert in any variant. The pattern ran against clinical severity: among the three scenarios with an identified method, only one of six triggered the banner. The guardrail fired more reliably when patients had not named a means of self-harm.