Search papers, labs, and topics across Lattice.
This paper investigates the safety vulnerabilities of self-evolving large language model (LLM) agents, revealing that benign experiences can collectively undermine an agent's safety when reused across sessions. The authors introduce EvoBreak, an innovative attack method that exploits the composition of these experiences through a series of individually harmless tasks, ultimately reformulating queries to activate harmful interactions. Experimental results show that EvoBreak significantly outperforms traditional memory attacks while maintaining a facade of benignness, highlighting a critical and overlooked risk in the deployment of self-evolving agents.
Benign experiences in self-evolving LLMs can be weaponized, revealing a hidden attack surface that undermines safety guarantees.
Self-evolving large language model agents improve their capabilities by distilling interaction trajectories into persistent experiences. Yet this mechanism introduces a new safety risk: experiences that are benign in isolation may jointly weaken an agent's safety boundary when accumulated and reused across sessions. Existing memory attacks typically require direct memory access or induce explicitly malicious records, limiting their stealthiness and applicability. We propose EvoBreak, an experience-conditioned sequential attack that operates through individually benign attack-stage tasks and induced experiences. EvoBreak repeatedly observes the experiences distilled by the victim, identifies uncovered target-relevant requirements, and adaptively acquires complementary experiences before reformulating the final query to activate them jointly. To support training, we introduce BreakGym, a structure-first synthesis pipeline that generates decomposable safety-sensitive targets with diverse dependency structures. EvoBreak is optimized using rejection-sampling supervised fine-tuning and Hint-guided GRPO. Experiments across self-evolving frameworks, victim backbones, pre-evolution domains, and safety benchmarks demonstrate that EvoBreak consistently outperforms existing attacks while maintaining high benignness. These results reveal benign experience composition as a persistent attack surface in self-evolving agents.
