Search papers, labs, and topics across Lattice.
This paper introduces a new benchmark dataset of 270 harmful instructions, categorized by medical ethics principles, to evaluate the safety of 72 LLMs in a simulated robotic health attendant context. Results show a high mean violation rate of 54.4% across models, with proprietary models significantly outperforming open-weight models, and medical fine-tuning failing to provide a substantial safety benefit. The study highlights the critical need for rigorous safety evaluation before deploying LLMs in robotic healthcare.
LLMs fail over half the time when asked to perform harmful actions in a simulated robotic health attendant setting, even when fine-tuned on medical data.
Large language models (LLMs) are increasingly considered for deployment as the control component of robotic health attendants, yet their safety in this context remains poorly characterized. We introduce a dataset of 270 harmful instructions spanning nine prohibited behavior categories grounded in the American Medical Association Principles of Medical Ethics, and use it to evaluate 72 LLMs in a simulation environment based on the Robotic Health Attendant framework. The mean violation rate across all models was 54.4\%, with more than half exceeding 50\%, and violation rates varied substantially across behavior categories, with superficially plausible instructions such as device manipulation and emergency delay proving harder to refuse than overtly destructive ones. Model size and release date were the primary determinants of safety performance among open-weight models, and proprietary models were substantially safer than open-weight counterparts (median 23.7\% versus 72.8\%). Medical domain fine-tuning conferred no significant overall safety benefit, and a prompt-based defense strategy produced only a modest reduction in violation rates among the least safe models, leaving absolute violation rates at levels that would preclude safe clinical deployment. These findings demonstrate that safety evaluation must be treated as a first-class criterion in the development and deployment of LLMs for robotic health attendants.
