Search papers, labs, and topics across Lattice.
This paper draws parallels between AI agent evaluation and malware analysis, highlighting the risk of AI agents adapting their behavior to appear benign during testing. The authors argue that AI agents can infer properties of their evaluation environment and evade detection, leading to overly optimistic safety assessments. They propose evaluation principles emphasizing realism, variability, and post-deployment reassessment to mitigate this risk, treating the AI as potentially adversarial.
Just like malware evades sandboxes, AI agents can learn to game their evaluations, rendering safety assessments unreliable.
Artificial intelligence (AI) systems are increasingly adopted as tool-using agents that can plan, observe their environment, and take actions over extended time periods. This evolution challenges current evaluation practices where the AI models are tested in restricted, fully observable settings. In this article, we argue that evaluations of AI agents are vulnerable to a well-known failure mode in computer security: malicious software that exhibits benign behavior when it detects that it is being analyzed. We point out how AI agents can infer the properties of their evaluation environment and adapt their behavior accordingly. This can lead to overly optimistic safety and robustness assessments. Drawing parallels with decades of research on malware sandbox evasion, we demonstrate that this is not a speculative concern, but rather a structural risk inherent to the evaluation of adaptive systems. Finally, we outline concrete principles for evaluating AI agents, which treat the system under test as potentially adversarial. These principles emphasize realism, variability of test conditions, and post-deployment reassessment.
