Search papers, labs, and topics across Lattice.
This paper introduces AGENTSCOPE, a neuro-symbolic framework designed to diagnose failures in LLM agents by abstracting their behaviors into structured representations. By employing LLM-guided reasoning in conjunction with newly defined neural invariants, AGENTSCOPE effectively identifies failure steps and their types within complex agent trajectories. The method demonstrates significant improvements in fault localization and attribution accuracy on both existing and novel datasets, marking a substantial advancement over current diagnostic techniques.
AGENTSCOPE reveals that integrating structured behavior abstractions with LLM reasoning can dramatically enhance the reliability of diagnosing agent failures.
With the proliferation of LLM agents, the ability to understand and diagnose failures in agents is essential to achieving superior effectiveness and trustworthiness. As agent failures often manifest via long and complex trajectories, manually finding the needles in the haystack is untenable. However, traditional diagnosis techniques for software bugs can hardly address LLM agent failures, while completely relying on LLMs as the judge yields unreliable diagnosis results. To overcome these challenges, this paper presents AGENTSCOPE, a new neuro-symbolic approach for agent failure mode diagnosis. The key principle of AGENTSCOPE is to abstract agent behavior, based on its trajectories, into structured representations. Furthermore, AGENTSCOPE introduces the concept of neural invariants to specify agent behavior properties. AGENTSCOPE leverages LLM-guided reasoning atop the structured representation against neural invariants to pinpoint both the failure step and its type in the trajectory. We show the effectiveness of AGENTSCOPE on publicly available agent failure datasets (Who&When) and a more comprehensive dataset created by us (AgentErrata), where AGENTSCOPE significantly outperforms the current state of the art in fault localization and attribution accuracy. Our work shows that integrating structured abstractions with LLM-guided reasoning enables effective, reliable, and interpretable diagnosis for agent failures.
