Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces HealthBench-Psych, a curated mental health subset of OpenAI's HealthBench, specifically designed to evaluate the performance of large language models (LLMs) in the mental health domain. By screening 5,000 physician-rubric conversations, the authors identified 610 relevant interactions through a transparent LLM-applied rubric and validated them with clinician reviews. The evaluation revealed a statistically tied performance among frontier models, highlighting measurable refusal behavior in some models and consistent rankings across different judges, thus providing a valuable resource for assessing LLMs in mental health applications.
A new benchmark reveals that leading LLMs exhibit measurable refusal behavior in mental health contexts, challenging assumptions about their reliability in sensitive applications.
General-purpose health benchmarks increasingly anchor claims about LLM medical performance, but they are not always resolved by clinical specialty, making domain-specific performance hard to isolate. Mental health is of acute public-health concern as millions of people turn to LLMs for psychological support, and most existing evaluations are bespoke academic benchmarks that are difficult to integrate into developer workflows. We introduce HealthBench-Psych and HealthBench-Psych-Hard. We screened HealthBench's 5,000 physician-rubric conversations for mental-health relevance with a transparent LLM-applied rubric, then validated the subset through two rounds of blinded clinician review with concealed known-exclude controls, yielding 610 conversations (12.2% of the corpus). Evaluating 20 frontier and open models under a cross-vendor panel of three LLM judges, we find a statistically tied frontier cluster, measurable refusal behavior in two models, and near-identical rankings across judges ($\tau \ge 0.92$). We release the subset, pipeline, model responses, grades, and analysis code as a reusable resource.