Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
AgentSafeEvo employs an adversarial co-evolutionary reinforcement learning framework to enhance LLM agent safety by simulating an attacker-defender game where the attacker crafts adversarial tasks and the defender learns to execute them safely. This approach specifically targets vulnerabilities in tool use and environment interaction, going beyond language-level safety. Experiments show AgentSafeEvo significantly improves adversarial robustness without compromising general agent capabilities.
LLM agents can be made robust to complex, multi-turn adversarial attacks on tool use through a co-evolutionary game, outperforming prior methods on safety and capability benchmarks.
Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker-defender game spanning diverse tasks and attack types. The attacker learns to construct adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on agent safety and general capability benchmarks demonstrate that our method improves adversarial robustness while maintaining general agent capabilities.