Search papers, labs, and topics across Lattice.
This paper identifies the "Severance Problem," where language models fail to represent the user beyond the immediate context, leading to issues like sycophancy and hallucination. By introducing the Severance Schema, which incorporates structured ignorance about the user across various dimensions, the authors demonstrate that models can significantly improve their performance in personal assistant tasks. Empirical results show that models utilizing this schema reduce undesirable behaviors and enhance their ability to ask clarifying questions when user information is incomplete.
Language models can drastically reduce sycophancy and hallucination by acknowledging what they don鈥檛 know about users.
Personal AI assistants have attracted significant interest for their potential to enhance everyday life by automating routine tasks, supporting consequential decisions, and assisting with everyday personal matters. Yet despite rapid recent technical advances, these assistants continue to exhibit undesirable behaviors, such as sycophancy, overconfidence, and hallucination. We argue that these failures stem from a fundamental limitation: language models lack an explicit representation of the person beyond the context they are given, which we term as the \textbf{Severance Problem}. Even with rich personal context and strong commonsense reasoning capabilities from the backbone model, current AI assistants fail to represent what remains unknown about the user. We propose a simple solution: incorporating structured ignorance into the language model context via the \textbf{Severance Schema}, which explicitly outlines dimensions along which the model lacks knowledge about the user, including physicality, temporality, consequences, continuity, multiplicity, and interiority. Empirically, across five model families, with the Severance Schema, the assistant consistently reduces sycophancy, harmful advice, and hallucination. Notably, models with the schema ask clarifying questions when information about the user is missing, rather than confidently extrapolating from incomplete user information.
