Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces AgentCanary, a novel security evaluation framework designed for autonomous AI agents operating in real executable environments, addressing critical gaps in existing evaluation methods. By implementing a comprehensive risk taxonomy and a high-fidelity environment, AgentCanary enables agents to interact with real tools and assess their performance across multi-dimensional metrics. The evaluation of various frontier models reveals significant vulnerabilities, with agents frequently failing to detect adversarial attacks, highlighting the urgent need for improved security measures in AI systems.
Current autonomous AI agents are alarmingly unprepared for real-world adversarial attacks, often missing critical vulnerabilities in dynamic environments.
Autonomous AI agents have driven the transition from conversation to task execution, shifting security failures from textual deception to system compromise. Although security evaluation is crucial for proactive risk prevention, prior work is constrained by fundamental bottlenecks, including fragmented risk coverage, static or low-fidelity execution environments, and single-dimensional and coarse-grained assessment metrics. To address these challenges, we propose AgentCanary, a comprehensive security evaluation framework for autonomous AI agents. AgentCanary provides a systematic solution along three contributions. First, comprehensive risk coverage: we introduce an orthogonal Entry $\times$ Impact risk taxonomy that decouples how adversarial influence enters the agent from what harm it ultimately causes, and instantiate it as a scenario-aligned task suite spanning realistic deployment workflows. Second, a high-fidelity real executable environment: rather than static Q&A or mocked tool responses, agents interact with real tools against dynamically provisioned task artifacts, with persistent state across multi-step interactions that naturally supports long-horizon attack evaluation. Third, trajectory-grounded multi-dimensional evaluation: evaluation consumes the full agent trajectory rather than the reply text or a single tool call, enabling decomposed scoring along three orthogonal dimensions, Outcome Safety, Security Awareness, and Task Utility. We evaluate a broad set of frontier models on AgentCanary against multiple established adversarial attack methods across three agent frameworks. The results reveal that current agents often fail to recognize the attacks they face, particularly under compromised skills, persistent state, and long-horizon execution attacks, and provide a systematic baseline for developing more reliable and secure agent systems.
Tsinghua AI