Search papers, labs, and topics across Lattice.
This study introduces CARE-Bench, a novel benchmark designed to evaluate the performance of patient-facing medical LLMs in triaging symptoms through a structured four-label action task. By analyzing 11 models across 500 cases, the research highlights that while prompting can enhance model performance, significant errors persist, particularly in recommending actions prematurely without sufficient patient information. The findings underscore the complexity of patient-facing triage and the necessity for rigorous evaluation of action timing before deploying these models in clinical settings.
Despite improvements from prompting, patient-facing LLMs still misjudge 66.5% of the time when more information is needed before recommending care.
Patient-facing medical LLMs and agents increasingly answer symptom questions before clinician contact, where the key safety question is what action the user should take next. We introduce CARE-Bench, a source-grounded benchmark that evaluates sequential patient-facing triage as a four-label per-turn current-action task. CARE-Bench contains 500 cases and 1,059 evaluated patient-disclosure prefixes reconstructed from medical dialogue, consultation, and follow-up-question sources. We evaluate 11 models on 269 held-out rounds under unprompted and minimally prompted open-ended protocols, using a fixed GPT-5.5 mapper to code each response into the four-label action space. Unprompted macro-F1 remains low, ranging from 31.2 to 50.4. Prompting improves 10 of 11 models, with prompted macro-F1 ranging from 46.9 to 63.4, but substantial threshold errors remain. Prompted models often recommend care before needed clarification is obtained; when the correct action was to ask for more information, only 33.5% of prompted outputs preserved the step. The persistence of these errors after prompting suggests that patient-facing triage is not a simple prompting problem and supports explicit evaluation of action timing before deployment.
