Search papers, labs, and topics across Lattice.
Microsoft Research
MIT CSAIL
Stanford HAIAnt GroupBJUTHKUHKUSTHunanInstitute of Science TokyoOxfordUNCXJTUZJUJun 1, 2026arXiv:2606.02302This paper introduces SeClaw, a novel framework that synthesizes security tasks for evaluating autonomous LLM agents in stateful environments, addressing the inadequacies of existing benchmarks that focus on final outcomes rather than execution processes. By leveraging specification-driven task synthesis, SeClaw enables the scalable creation of security tasks derived from structured risk specifications, while also providing a standardized testbed for assessing agent behavior in various safety-risk scenarios. The key finding reveals that SeClaw effectively enhances the coverage of emerging threats and allows for trajectory-aware evaluations of unsafe actions, thereby improving the diagnosis and comparison of security failures in autonomous agents.
SeClaw reveals that existing benchmarks fall short in capturing the complexities of agent behavior, enabling a more nuanced evaluation of security risks in autonomous systems.
Autonomous LLM agents increasingly operate in stateful environments where they access tools, files, memory, and external services. While such capabilities enable complex real-world workflows, they also introduce security risks that are difficult to capture with existing evaluations. Current agent security benchmarks often rely on manually curated tasks, provide limited coverage of emerging threats, and focus primarily on final outcomes rather than the execution processes that lead to unsafe behavior. We introduce SeClaw, a framework that combines specification-driven security task synthesis with execution-based security evaluation for Autonomous agents. Spec-driven security task synthesis enables scalable and controllable construction of security tasks from structured risk specifications, while SeClaw docker provides a standardized testbed for evaluating agent behavior under diverse safety-risk scenarios. The benchmark covers risks arising from resources, user tasks, environments, and intrinsic agent behaviors, and supports trajectory-aware assessment of unsafe actions beyond final responses. By bridging systematic task synthesis and reproducible security evaluation, SeClaw provides a practical foundation for measuring, diagnosing, and comparing security failures in autonomous LLM agents. The code is available at https://github.com/seclaw-eval/seclaw-eval.
