Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates a phenomenon termed Protective Capacity Hallucination (PCH), where large language models (LLMs) assert capabilities they do not possess when tasked with protecting vulnerable users. Through a comprehensive analysis involving eight LLMs and 13,600 interactions, the researchers found that PCH is influenced by the severity of the situation and the format of the interaction, with multi-party dialogues leading to exaggerated claims of capability. Notably, in contexts like intimate-partner conflict, where safety alignment is emphasized, LLMs did not exhibit PCH despite the higher stakes, highlighting a critical gap in deployment design regarding capability boundaries.
LLMs may falsely claim protective actions they cannot perform, revealing a dangerous gap between their intended roles and actual capabilities.
When cast as the protector of a vulnerable user yet given no explicit capability boundary, a large language model (LLM) may respond not by acknowledging its limits but by claiming to have taken -- or to be taking -- a real-world protective action it cannot perform, such as contacting emergency services or administering care. We term this phenomenon Protective Capacity Hallucination (PCH): a self-referential misattribution in which a model, acting in a protective role, asserts physical or institutional agency exceeding its affordances as a language model. In a three-phase study spanning eight LLMs and 13{,}600 sessions, we find PCH jointly gated by situational severity and interactional format: multi-party dialogic input drives it toward ceiling in most models across ordinary service domains, whereas in intimate-partner conflict -- a domain explicitly covered by safety alignment -- it remains at floor in all eight models despite greater physical severity. We interpret PCH as the signature of a deployment-design gap between role assignment and capability-boundary specification: a by-product of partial alignment in which a universally trained pressure to help outruns a domain-selective specification of how to help. Because suppression tracks alignment coverage rather than severity, deployment-side specification of capability boundaries emerges as a general mitigation target.