Search papers, labs, and topics across Lattice.
This paper introduces MuPPET, a benchmark designed to evaluate the contextual privacy of large language model (LLM) assistants in multi-party conversations, highlighting the unique privacy risks posed in these settings. The authors demonstrate that LLMs leak significantly more private information in multi-party environments compared to one-on-one interactions, revealing vulnerabilities in both frontier models and smaller open-weight models commonly used for local deployments. Furthermore, they find that current privacy defenses are inadequate, providing only partial protection while degrading model utility and failing to address the core issue of party tracking.
Multi-party conversations expose LLMs to a staggering increase in privacy leaks, with models revealing sensitive information far more than previously understood.
LLM agents are increasingly deployed in multi-party environments, handling sensitive personal data on behalf of individual users, for instance in group chats. When such an agent discloses private information, it reaches every group member at once. This risk is structurally harder to control than in one-to-one settings, as every piece of private information must be appropriate for every recipient in the group. Yet all existing contextual privacy benchmarks consider only single-interlocutor settings, leaving multi-party privacy risks unmeasured. We introduce MuPPET (Multi-Party Privacy Exposure Testing), a benchmark for contextual privacy in multi-party conversations. Our experiments show that models leak substantially more in multi-party settings than one-to-one evaluations suggest. Frontier models are vulnerable, and smaller open-weights models, often preferred for local deployment with sensitive data, even more so. Existing contextual privacy defences offer only partial protection, degrade utility, and do not resolve the underlying party-tracking problem.
