Search papers, labs, and topics across Lattice.
This paper introduces UMPeek, a novel black-box attack that infers hidden user models from the behavior of personalized LLM agents, highlighting the vulnerabilities in privacy-preserving designs. By leveraging hypothesis-guided adaptive probing, UMPeek effectively reconstructs user information from the choices made by the agents, even when direct memory access is restricted. The extensive evaluation demonstrates that UMPeek significantly outperforms existing attacks, revealing that the mere absence of source records does not ensure semantic privacy in personalized AI systems.
UMPeek reveals that even when memory access is restricted, personalized LLM agents can still leak sensitive user information through their decision-making patterns.
Recent personalized LLM agents increasingly transform information retained in memory into compressed or structured representations, which we call user models, to guide later decisions. When source wording is removed from the state reachable through the ordinary interface, these models are commonly treated as more privacy-preserving because direct memory-extraction attacks lose the text they target. Yet we argue that user models expose a new attack surface because an attacker can still recover the private information from the personalized choices they shape, even when source records and backend state remain inaccessible. We therefore introduce UMPeek, a black-box attack based on hypothesis-guided adaptive probing to infer such hidden user model. It forms hypotheses from choices left open by a request, switches among ordinary follow-up tasks, and retains only claims supported and not contradicted by visible behavior. We conduct an extensive benchmark evaluation across diverse personalization tasks and user-model backends against existing attacks. We further validate UMPeek in real-world systems using information confirmed to be retained, and we evaluate defenses against its adaptive probing. Overall, UMPeek outperforms existing attacks in both benchmark and real-world comparisons and continues to recover user information under response-level defenses, showing that keeping records and backend state inaccessible does not guarantee semantic privacy when retained information shapes visible behavior.
