Search papers, labs, and topics across Lattice.
This paper introduces AgentCompass, a unified evaluation infrastructure designed to enhance the assessment of Large Language Model (LLM)-based autonomous agents. By addressing the fragmentation and rigidity of existing evaluation pipelines, AgentCompass allows for flexible configurations through its modular design, which includes components for Benchmark, Harness, and Environment. Key results demonstrate its capability to support over 20 benchmarks across five dimensions, facilitating scalable and reproducible research while enabling detailed analysis of agent performance and failure modes.
AgentCompass transforms agent evaluation by providing a modular, open-source framework that supports over 20 benchmarks and enables nuanced failure analysis.
As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical. However, current evaluation pipelines remain highly fragmented and tightly coupled, hindering reproducibility and causing redundant engineering. To address this, we introduce AgentCompass, an open-source, lightweight, and extensible infrastructure for evaluating LLM-based agents. AgentCompass organizes the evaluation process around three independent components, namely Benchmark, Harness, and Environment, thereby enabling flexible configurations without requiring the reimplementation of complex execution logic. Furthermore, it features a fault-tolerant asynchronous runtime and comprehensive trajectory analysis tools to transparently diagnose nuanced failure modes like reward-hacking. Natively supporting over 20 benchmarks across five capability dimensions, AgentCompass provides the community with a scalable and reproducible infrastructure for advancing agent research.
